The way to do it is to use client-side event to get to the respective edit field (typically textbox) and attach the custom 3rd party edit control to it. While the principles are the same for both inline and edit dialog modes, some difference aplply.

Inline Mode

For inline mode to work, you need to use the client-side API of the grid to set the row into inline editing (more in the Inline Editing section of the help). When you do so, also set a callback function for the editRow method that executes when the row is ready and in edit mode. You can use the callback to hook custom code and enable your 3rd party editor. Example:

<trirand:JQGrid ID="JQGrid1" runat="server" Width="600px" ShowToolBar="True" OnRowEditing="JQGrid1_RowEditing">         
     <Columns>
         <trirand:JQGridColumn DataField="EmployeeID" Editable="false" PrimaryKey="true" Width="50"/>                
         <trirand:JQGridColumn DataField="FirstName" Editable="true" />
         <trirand:JQGridColumn DataField="LastName" Editable="true" />
         <trirand:JQGridColumn DataField="BirthDate" Editable="true" DataFormatString="{0:yyyy-MM-dd}" />
     </Columns>
     <ClientSideEvents OnClientRowSelect="editRow" />
</trirand:JQGrid>

<script type="text/javascript">                     

           var lastSelection;

           function editRow(id) {
               if (id && id !== lastSelection) {
                   var grid = jQuery("#<%= JQGrid1.ClientID %>");
                   grid.restoreRow(lastSelection);
                   grid.editRow(id, true, pickdates);
                   lastSelection = id;
               }
           }

           function pickdates(id) {
               jQuery("#" + id + "_BirthDate", "#<%= JQGrid1.ClientID %>").datepicker({
                   dateFormat: "yy-mm-dd",
                   onSelect: function(dateText, inst) {
                       var grid = jQuery("#<%= JQGrid1.ClientID %>");
                       grid.saveRow(id);
                   }
               });               
           }
       </script>

Note, that the callback function is the 3rd parameter of the editRow method. Also, the ID of the editing control is produced by ID key of the row + "_" + the datafield of the column (see the pickdates method for details).