PSARC/2009/331 IP Datapath Refactoring

PSARC/2008/522 EOF of 2001/070 IPsec HW Acceleration support
PSARC/2009/495 netstat -r flags for blackhole and reject routes
PSARC 2009/496 EOF of XRESOLV
PSARC/2009/494 IP_DONTFRAG socket option
PSARC/2009/515 fragmentation controls for ping and traceroute
6798716 ip_newroute delenda est
6798739 ARP and IP are too separate
6807265 IPv4 ip2mac() support
6756382 Please remove Venus IPsec HWACCEL code
6880632 sendto/sendmsg never returns EHOSTUNREACH in Solaris
6748582 sendmsg() return OK, but doesn't send message using IPv4-mapped x IPv6 addr
1119790 TCP and path mtu discovery
4637227 should support equal-cost multi-path (ECMP)
5078568 getsockopt() for IPV6_PATHMTU on a non-connected socket should not succeed
6419648 "AR* contract private note" should be removed as part of ATM SW EOL
6274715 Arp could keep the old entry in the cache while it waits for an arp response
6605615 Remove duplicated TCP/IP opt_set/opt_get code; use conn_t
6874677 IP_TTL can be used to send with ttl zero
4034090 arp should not let you delete your own entry
6882140 Implement IP_DONTFRAG socket option
6883858 Implement ping -D option; traceroute -F should work for IPv6 and shared-IP zones
1119792 TCP/IP black hole detection is broken on receiver
4078796 Directed broadcast forwarding code has problems
4104337 restrict the IPPROTO_IP and IPPROTO_IPV6 options based on the socket family
4203747 Source address selection for source routed packets
4230259 pmtu is increased every ip_ire_pathmtu_interval timer value.
4300533 When sticky option ipv6_pktinfo set to bogus address subsequent connect time out
4471035 ire_delete_cache_gw is called through ire_walk unnecessarily
4514572 SO_DONTROUTE socket option doesn't work with IPv6
4524980 tcp_lookup_ipv4() should compare the ifindex against tcpb->tcpb_bound_if
4532714 machine fails to switch quickly among failed default routes
4634219 IPv6 path mtu discovery is broken when using routing header
4691581 udp broadcast handling causes too many replicas
4708405 mcast is broken on machines when all interfaces are IFF_POINTOPOINT
4770457 netstat/route: source address of interface routes pretends to be gateway address
4786974 use routing table to determine routes/interface for multicast
4792619 An ip_fanout_udp_ipc_v6() routine might lead to some simpler code
4816115 Nuke ipsec_out_use_global_policy
4862844 ipsec offload corner case
4867533 tcp_rq and tcp_wq are redundant
4868589 NCEs should be shared across an IPMP group
4872093 unplumbing an improper virtual interface panics in ip_newroute_get_dst_ill()
4901671 FireEngine needs some cleanup
4907617 IPsec identity latching should be done before sending SYN-ACK
4941461 scopeid and IPV6_PKTINFO with UDP/ICMP connect() does not work properly
4944981 ip does nothing with IP6I_NEXTHOP
4963353 IPv4 and IPv6 proto fanout codes could be brought closer
4963360 consider passing zoneid using ip6i_t instead of ipsec_out_t in NDP
4963734 new ip6_asp locking is used incorrectly in ip_newroute_v6()
5008315 IPv6 code passes ip6i_t to IPsec code instead of ip6_t
5009636 memory leak in ip_fanout_proto_v6()
5092337 tcp/udp option handling can use some cleanup
5035841 Solaris can fail to create a valid broadcast ire
5043747 ar_query_xmit: Could not find the ace
5051574 tcp_check_policy is missing some checks
6305037 full hardware checksum is discarded when there're more than 2 mblks in the chain
6311149 ip.c needs to be put through a woodchipper
4708860 Unable to reassemble CGTP fragmented multicast packets
6224628 Large IPv6 packets with IPsec protection sometimes have length mismatch.
6213243 Solaris does not currently support Dead Gateway Detection
5029091 duplicate code in IP's input path for TCP/UDP/SCTP
4674643 through IPv6 CGTP routes, the very first packet is sent only after a while
6207318 Multiple default routes do not round robin connections to routers.
4823410 IP has an inconsistent view of link mtu
5105520 adding interface route to down interface causes ifconfig hang
5105707 advanced sockets API introduced some dead code
6318399 IP option handling for icmp and udp is too complicated
6321434 Every dropped packet in IP should use ip_drop_packet()
6341693 ifconfig mtu should operate on the physical interface, not individual ipif's
6352430 The credentials attached to an mblk are not particularly useful
6357894 uninitialised ipp_hoplimit needs to be cleaned up.
6363568 ip_xmit_v6() may be missing IRE releases in error cases
6364828 ip_rput_forward needs a makeover
6384416 System panics when running as multicast forwarder using multicast tunnels
6402382 TX: UDP v6 slowpath is not modified to handle mac_exempt conns
6418413 assertion failed ipha->ipha_ident == 0||ipha->ipha_ident == 0xFFFF
6420916 assertion failures in ipv6 wput path
6430851 use of b_prev to store ifindex is not 100% safe
6446106 IPv6 packets stored in nce->nce_qd_mp will be sent with incorrect tcp/udp checksums
6453711 SCTP OOTB sent as if genetated by global zone
6465212 ARP/IP merge should remove ire_freemblk.esballoc
6490163 ip_input() could misbehave if the first mblk's size is not big enough
6496664 missing ipif_refrele leads to reference leak and deferred crash in ip_wput_ipsec_out_v6
6504856 memory leak in ip_fanout_proto_v6() when using link local outer tunnel addresses
6507765 IRE cache hash function performs badly
6510186 IP_FORWARD_PROG bit is easily overlooked
6514727 cgtp ipv6 failure on snv54
6528286 MULTIRT (CGTP) should offload checksum to hardware
6533904 SCTP: doesn't support traffic class for IPv6
6539415 TX: ipif source selection is flawed for unlabeled gateways
6539851 plumbed unworking nic blocks sending broadcast packets
6564468 non-solaris SCTP stack over rawip socket: netstat command counts rawipInData not rawipOutDatagrams
6568511 ipIfStatsOutDiscards not bumped when discarding an ipsec packet on the wrong NIC
6584162 tcp_g_q_inactive() makes incorrect use of taskq_dispatch()
6603974 round-robin default with many interfaces causes infinite temporary IRE thrashing
6611750 ilm_lookup_ill_index_v4 was born an orphan
6618423 ip_wput_frag_mdt sends out packets that void pfhooks
6620964 IRE max bucket count calculations performed in ip_ire_init() are flawed
6626266 various _broadcasts seem redundant
6638182 IP_PKTINFO + SO_DONTROUTE + CIPSO IP option == panic
6647710 IPv6 possible DoS vulnerability
6657357 nce should be kmem_cache alloc'ed from an nce_cache.
6685131 ilg_add -> conn_ilg_alloc interacting with conn_ilg[] walkers can cause panic.
6730298 adding 0.0.0.0 key with mask != 0 causes  'route delete default' to fail
6730976 vni and ipv6 doesn't quite work.
6740956 assertion failed: mp->b_next == 0L && mp->b_prev == 0L in nce_queue_mp_common()
6748515 BUMP_MIB() is occasionally done on the wrong ill
6753250 ip_output_v6() `notv6' error path has an errant ill_refrele()
6756411 NULL-pointer dereference in ip_wput_local()
6769582 IP must forward packet returned from FW-HOOK
6781525 bogus usesrc usage leads directly to panic
6422839 System paniced in ip_multicast_loopback due to NULL pointer dereference
6785521 initial IPv6 DAD solicitation is dropped in ip_newroute_ipif_v6()
6787370 ipnet devices not seeing forwarded IP packets on outgoing interface
6791187 ip*dbg() calls in ip_output_options() claim to originate from ip_wput()
6794047 nce_fp_mp prevents sharing of NCEs across an IPMP group
6797926 many unnecessary ip0dbg() in ip_rput_data_v6
6846919 Packet queued for ND gets sent in the clear.
6856591 ping doesn't send packets with DF set
6861113 arp module has incorrect dependency path for hook module
6865664 IPV6_NEXTHOP does not work with TCP socket
6874681 No ICMP time exceeded when a router receives packet with ttl = 0
6880977 ip_wput_ire() uses over 1k of stack
6595433 IPsec performance could be significantly better when calling hw crypto provider synchronously
6848397 ifconfig down of an interface can hang.
6849602 IPV6_PATHMTU size issue for UDP
6885359 Add compile-time option for testing pure IPsec overhead
6889268 Odd loopback source address selection with IPMP
6895420 assertion failed: connp->conn_helper_info == NULL
6851189 Routing-related panic occurred during reboot on T2000 system running snv_117
6896174 Post-async-encryption, AH+ESP packets may have misinitialized ipha/ip6
6896687 iptun presents IPv6 with an MTU < 1280
6897006 assertion failed: ipif->ipif_id != 0 in ip_sioctl_slifzone_restart

13 files changed, 209 insertions, 1090 deletions

diff --git a/usr/src/uts/common/io/dld/dld_proto.c b/usr/src/uts/common/io/dld/dld_proto.c
index 338a1c96d0..79b88ca659 100644
--- a/usr/src/uts/common/io/dld/dld_proto.c
+++ b/usr/src/uts/common/io/dld/dld_proto.c
@@ -1478,7 +1478,7 @@ dld_capab_lso(dld_str_t *dsp, void *data, uint_t flags)
 			lso->lso_flags = 0;
 			/* translate the flag for mac clients */
 			if ((mac_lso.lso_flags & LSO_TX_BASIC_TCP_IPV4) != 0)
-				lso->lso_flags |= DLD_LSO_TX_BASIC_TCP_IPV4;
+				lso->lso_flags |= DLD_LSO_BASIC_TCP_IPV4;
 			dsp->ds_lso = B_TRUE;
 			dsp->ds_lso_max = lso->lso_max;
 		} else {
diff --git a/usr/src/uts/common/io/ib/clients/rds/rds_opt.c b/usr/src/uts/common/io/ib/clients/rds/rds_opt.c
index 902d838ff4..639bb28bcc 100644
--- a/usr/src/uts/common/io/ib/clients/rds/rds_opt.c
+++ b/usr/src/uts/common/io/ib/clients/rds/rds_opt.c
@@ -19,7 +19,7 @@
  * CDDL HEADER END
  */
 /*
- * Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
+ * Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
  * Use is subject to license terms.
  */
 
@@ -29,9 +29,9 @@
 #define	rds_max_buf 2097152
 opdes_t rds_opt_arr[] = {
 
-{ SO_TYPE,	SOL_SOCKET, OA_R, OA_R, OP_NP, OP_PASSNEXT, sizeof (int), 0 },
-{ SO_SNDBUF,	SOL_SOCKET, OA_RW, OA_RW, OP_NP, OP_PASSNEXT, sizeof (int), 0 },
-{ SO_RCVBUF,	SOL_SOCKET, OA_RW, OA_RW, OP_NP, OP_PASSNEXT, sizeof (int), 0 },
+{ SO_TYPE,	SOL_SOCKET, OA_R, OA_R, OP_NP, 0, sizeof (int), 0 },
+{ SO_SNDBUF,	SOL_SOCKET, OA_RW, OA_RW, OP_NP, 0, sizeof (int), 0 },
+{ SO_RCVBUF,	SOL_SOCKET, OA_RW, OA_RW, OP_NP, 0, sizeof (int), 0 },
 };
 
 /* ARGSUSED */
@@ -79,7 +79,7 @@ rds_opt_get(queue_t *q, t_scalar_t level, t_scalar_t name, uchar_t *ptr)
 int
 rds_opt_set(queue_t *q, uint_t optset_context, int level,
     int name, uint_t inlen, uchar_t *invalp, uint_t *outlenp,
-    uchar_t *outvalp, void *thisdg_attrs, cred_t *cr, mblk_t *mblk)
+    uchar_t *outvalp, void *thisdg_attrs, cred_t *cr)
 {
 	int	*i1 = (int *)(uintptr_t)invalp;
 	boolean_t checkonly;
@@ -187,7 +187,6 @@ optdb_obj_t rds_opt_obj = {
 	rds_opt_default,	/* RDS default value function pointer */
 	rds_opt_get,		/* RDS get function pointer */
 	rds_opt_set,		/* RDS set function pointer */
-	B_TRUE,			/* RDS is tpi provider */
 	RDS_OPT_ARR_CNT,	/* RDS option database count of entries */
 	rds_opt_arr,		/* RDS option database */
 	RDS_VALID_LEVELS_CNT,	/* RDS valid level count of entries */
diff --git a/usr/src/uts/common/io/ib/clients/rds/rdsddi.c b/usr/src/uts/common/io/ib/clients/rds/rdsddi.c
index a4a9c6c8e0..13a1d4bf75 100644
--- a/usr/src/uts/common/io/ib/clients/rds/rdsddi.c
+++ b/usr/src/uts/common/io/ib/clients/rds/rdsddi.c
@@ -654,11 +654,9 @@ rds_wput_other(queue_t *q, mblk_t *mp)
 			}
 			if (((union T_primitives *)(uintptr_t)rptr)->type ==
 			    T_SVR4_OPTMGMT_REQ) {
-				(void) svr4_optcom_req(q, mp, cr, &rds_opt_obj,
-				    B_FALSE);
+				svr4_optcom_req(q, mp, cr, &rds_opt_obj);
 			} else {
-				(void) tpi_optcom_req(q, mp, cr, &rds_opt_obj,
-				    B_FALSE);
+				tpi_optcom_req(q, mp, cr, &rds_opt_obj);
 			}
 			return;
 		case T_CONN_REQ:
diff --git a/usr/src/uts/common/io/ib/clients/rds/rdssubr.c b/usr/src/uts/common/io/ib/clients/rds/rdssubr.c
index 8e57cb783d..f9bbcd092f 100644
--- a/usr/src/uts/common/io/ib/clients/rds/rdssubr.c
+++ b/usr/src/uts/common/io/ib/clients/rds/rdssubr.c
@@ -19,12 +19,10 @@
  * CDDL HEADER END
  */
 /*
- * Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
+ * Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
  * Use is subject to license terms.
  */
 
-#pragma ident	"%Z%%M%	%I%	%E% SMI"
-
 #include <sys/ib/clients/rds/rds.h>
 #include <sys/ib/clients/rds/rds_kstat.h>
 
@@ -135,9 +133,9 @@ rds_init()
 	 * kstats
 	 */
 	rds_kstatsp = kstat_create("rds", 0,
-		"rds_kstat", "misc", KSTAT_TYPE_NAMED,
-		sizeof (rds_kstat) / sizeof (kstat_named_t),
-		KSTAT_FLAG_VIRTUAL | KSTAT_FLAG_WRITABLE);
+	    "rds_kstat", "misc", KSTAT_TYPE_NAMED,
+	    sizeof (rds_kstat) / sizeof (kstat_named_t),
+	    KSTAT_FLAG_VIRTUAL | KSTAT_FLAG_WRITABLE);
 	if (rds_kstatsp != NULL) {
 		rds_kstatsp->ks_lock = &rds_kstat_mutex;
 		rds_kstatsp->ks_data = (void *)&rds_kstat;
@@ -298,17 +296,14 @@ rds_fanout(ipaddr_t local_addr, ipaddr_t rem_addr,
 boolean_t
 rds_islocal(ipaddr_t addr)
 {
-	ire_t *ire;
 	ip_stack_t *ipst;
 
 	ipst = netstack_find_by_zoneid(GLOBAL_ZONEID)->netstack_ip;
 	ASSERT(ipst != NULL);
-
-	ire = ire_ctable_lookup(addr, NULL, IRE_LOCAL | IRE_LOOPBACK |
-	    IRE_BROADCAST, NULL, ALL_ZONES, NULL, MATCH_IRE_TYPE, ipst);
-	netstack_rele(ipst->ips_netstack);
-	if (ire == NULL)
+	if (ip_laddr_verify_v4(addr, ALL_ZONES, ipst, B_FALSE) == IPVL_BAD) {
+		netstack_rele(ipst->ips_netstack);
 		return (B_FALSE);
-	ire_refrele(ire);
+	}
+	netstack_rele(ipst->ips_netstack);
 	return (B_TRUE);
 }
diff --git a/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp.c b/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp.c
index 944e61a067..3bb7d3a98c 100644
--- a/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp.c
+++ b/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp.c
@@ -26,41 +26,28 @@
 #include <sys/types.h>
 #include <sys/ddi.h>
 #include <sys/sunddi.h>
-#include <sys/stropts.h>
-#include <sys/stream.h>
-#include <sys/strsun.h>
 #include <sys/strsubr.h>
 #include <sys/socket.h>
-#include <sys/stat.h>
 #include <net/if_arp.h>
 #include <net/if_types.h>
-#include <sys/file.h>
 #include <sys/sockio.h>
 #include <sys/pathname.h>
-#include <inet/arp.h>
-#include <sys/modctl.h>
 
 #include <sys/ib/mgt/ibcm/ibcm_arp.h>
 
 #include <sys/kstr.h>
-#include <sys/tiuser.h>
 #include <sys/t_kuser.h>
 
 extern char cmlog[];
 
-extern int ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
-    ibt_ip_addr_t *src_addr, ibcm_arp_pr_comp_func_t func);
-extern void ibcm_arp_pr_arp_ack(mblk_t *mp);
-extern void ibcm_arp_prwqn_delete(ibcm_arp_prwqn_t *wqnp);
+extern int ibcm_resolver_pr_lookup(ibcm_arp_streams_t *ib_s,
+    ibt_ip_addr_t *dst_addr, ibt_ip_addr_t *src_addr);
+extern void ibcm_arp_delete_prwqn(ibcm_arp_prwqn_t *wqnp);
 
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", datab))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", ibt_ip_addr_s))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", ibcm_arp_ip_t))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", ibcm_arp_ibd_insts_t))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", ibcm_arp_prwqn_t))
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", iocblk))
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", msgb))
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", queue))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", sockaddr_in))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", sockaddr_in6))
 
@@ -89,269 +76,6 @@ ibcm_ip_print(char *label, ibt_ip_addr_t *ipaddr)
 	}
 }
 
-/*
- * ibcm_arp_get_ibaddr_cb
- */
-static int
-ibcm_arp_get_ibaddr_cb(void *arg, int status)
-{
-	ibcm_arp_prwqn_t	*wqnp = (ibcm_arp_prwqn_t *)arg;
-	ibcm_arp_streams_t	*ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_get_ibaddr_cb(ib_s: %p wqnp: %p)",
-	    ib_s, wqnp);
-
-	mutex_enter(&ib_s->lock);
-	ib_s->status = status;
-	ib_s->done = B_TRUE;
-
-	IBTF_DPRINTF_L3(cmlog, "ibcm_arp_get_ibaddr_cb: SGID %llX:%llX "
-	    "DGID: %llX:%llX", wqnp->sgid.gid_prefix, wqnp->sgid.gid_guid,
-	    wqnp->dgid.gid_prefix, wqnp->dgid.gid_guid);
-
-	/* lock is held by the caller. */
-	cv_signal(&ib_s->cv);
-	mutex_exit(&ib_s->lock);
-	return (0);
-}
-
-/*
- * Lower read service procedure (messages coming back from arp/ip).
- * Process messages based on queue type.
- */
-static int
-ibcm_arp_lrsrv(queue_t *q)
-{
-	mblk_t *mp;
-	ibcm_arp_streams_t *ib_s = q->q_ptr;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_lrsrv(%p, ibd_s: 0x%p)", q, ib_s);
-
-	if (WR(q) == ib_s->arpqueue) {
-		while (mp = getq(q)) {
-			ibcm_arp_pr_arp_ack(mp);
-		}
-	}
-
-	return (0);
-}
-
-/*
- * Lower write service procedure.
- * Used when lower streams are flow controlled.
- */
-static int
-ibcm_arp_lwsrv(queue_t *q)
-{
-	mblk_t *mp;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_lwsrv(%p)", q);
-
-	while (mp = getq(q)) {
-		if (canputnext(q)) {
-			putnext(q, mp);
-		} else {
-			(void) putbq(q, mp);
-			qenable(q);
-			break;
-		}
-	}
-
-	return (0);
-}
-
-/*
- * Lower read put procedure. Arp/ip messages come here.
- */
-static int
-ibcm_arp_lrput(queue_t *q, mblk_t *mp)
-{
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_lrput(0x%p, db_type: %d)",
-	    q, DB_TYPE(mp));
-
-	switch (DB_TYPE(mp)) {
-		case M_FLUSH:
-			/*
-			 * Turn around
-			 */
-			if (*mp->b_rptr & FLUSHW) {
-				*mp->b_rptr &= ~FLUSHR;
-				qreply(q, mp);
-				return (0);
-			}
-			freemsg(mp);
-			break;
-		case M_IOCACK:
-		case M_IOCNAK:
-		case M_DATA:
-			/*
-			 * This could be in interrupt context.
-			 * Some of the ibt calls cannot be called in
-			 * interrupt context, so
-			 * put it in the queue and the message will be
-			 * processed by service proccedure
-			 */
-			(void) putq(q, mp);
-			qenable(q);
-			break;
-		default:
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_lrput: "
-			    "got unknown msg <0x%x>\n", mp->b_datap->db_type);
-			ASSERT(0);
-			break;
-	}
-
-	return (0);
-}
-
-/*
- * Streams write queue module info
- */
-static struct module_info ibcm_arp_winfo = {
-	0,		/* module ID number */
-	"ibcm",		/* module name */
-	0,		/* min packet size */
-	INFPSZ,
-	49152,		/* STREAM queue high water mark -- 49152 */
-	12		/* STREAM queue low water mark -- 12 */
-};
-
-/*
- * Streams lower write queue, for ibcm/ip requests.
- */
-static struct qinit ibcm_arp_lwinit = {
-	NULL,		/* qi_putp */
-	ibcm_arp_lwsrv,	/* qi_srvp */
-	NULL,		/* qi_qopen */
-	NULL,		/* qi_qclose */
-	NULL,		/* qi_qadmin */
-	&ibcm_arp_winfo,	/* module info */
-	NULL,		/* module statistics struct */
-	NULL,
-	NULL,
-	STRUIOT_NONE	/* stream uio type is standard uiomove() */
-};
-
-/*
- * Streams lower read queue: read reply messages from ibcm/ip.
- */
-static struct qinit ibcm_arp_lrinit = {
-	ibcm_arp_lrput,	/* qi_putp */
-	ibcm_arp_lrsrv,	/* qi_srvp */
-	NULL,		/* qi_qopen */
-	NULL,		/* qi_qclose */
-	NULL,		/* qi_qadmin */
-	&ibcm_arp_winfo,	/* module info */
-	NULL,		/* module statistics struct */
-	NULL,
-	NULL,
-	STRUIOT_NONE /* stream uio type is standard uiomove() */
-};
-
-
-static int
-ibcm_arp_link_driver(ibcm_arp_streams_t *ib_s, char *path, queue_t **q,
-    vnode_t **dev_vp)
-{
-	struct stdata *dev_stp;
-	vnode_t *vp;
-	int error;
-	queue_t *rq;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_link_driver: Enter: %s", path);
-
-	/* open the driver from inside the kernel */
-	error = vn_open(path, UIO_SYSSPACE, FREAD|FWRITE, 0, &vp,
-	    0, NULL);
-	if (error) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_arp_link_driver: "
-		    "vn_open('%s') failed\n", path);
-		return (error);
-	}
-	*dev_vp = vp;
-
-	dev_stp = vp->v_stream;
-	*q = dev_stp->sd_wrq;
-
-	VN_HOLD(vp);
-
-	rq = RD(dev_stp->sd_wrq);
-	RD(rq)->q_ptr = WR(rq)->q_ptr = ib_s;
-	setq(rq, &ibcm_arp_lrinit, &ibcm_arp_lwinit, NULL, QMTSAFE,
-	    SQ_CI|SQ_CO, B_FALSE);
-
-	return (0);
-}
-
-extern struct qinit strdata;
-extern struct qinit stwdata;
-
-/*
- * Unlink ip, ibcm, icmp6 drivers
- */
-/* ARGSUSED */
-static int
-ibcm_arp_unlink_driver(queue_t **q, vnode_t **dev_vp)
-{
-	vnode_t *vp = *dev_vp;
-	struct stdata *dev_stp = vp->v_stream;
-	queue_t *wrq, *rq;
-	int	rc;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_unlink_driver: Enter: 0x%p", q);
-
-	wrq = dev_stp->sd_wrq;
-	rq = RD(wrq);
-
-	disable_svc(rq);
-	wait_svc(rq);
-	flushq(rq, FLUSHALL);
-	flushq(WR(rq), FLUSHALL);
-
-	rq->q_ptr = wrq->q_ptr = dev_stp;
-
-	setq(rq, &strdata, &stwdata, NULL, QMTSAFE, SQ_CI|SQ_CO, B_TRUE);
-
-	if ((rc = VOP_CLOSE(vp, FREAD, 1, (offset_t)0, CRED(), NULL)) != 0) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_arp_unlink_driver: VOP_CLOSE "
-		    "failed %d\n", rc);
-	}
-	VN_RELE(vp);
-
-	return (0);
-}
-
-static int
-ibcm_arp_unlink_drivers(ibcm_arp_streams_t *ib_s)
-{
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_unlink_drivers(%p)", ib_s);
-
-	if (ib_s->arpqueue) {
-		(void) ibcm_arp_unlink_driver(&ib_s->arpqueue, &ib_s->arp_vp);
-	}
-
-	return (0);
-}
-
-/*
- * Link ip, ibtl drivers below ibtl
- */
-static int
-ibcm_arp_link_drivers(ibcm_arp_streams_t *ib_s)
-{
-	int	rc;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_link_drivers(%p)", ib_s);
-
-	if ((rc = ibcm_arp_link_driver(ib_s, "/dev/arp", &ib_s->arpqueue,
-	    &ib_s->arp_vp)) != 0) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_arp_link_drivers: "
-		    "ibcm_arp_link_driver failed: %d\n", rc);
-		return (rc);
-	}
-
-	return (0);
-}
 
 ibt_status_t
 ibcm_arp_get_ibaddr(ibt_ip_addr_t srcaddr, ibt_ip_addr_t destaddr,
@@ -370,21 +94,13 @@ ibcm_arp_get_ibaddr(ibt_ip_addr_t srcaddr, ibt_ip_addr_t destaddr,
 	mutex_init(&ib_s->lock, NULL, MUTEX_DEFAULT, NULL);
 	cv_init(&ib_s->cv, NULL, CV_DRIVER, NULL);
 
-	ret = ibcm_arp_link_drivers(ib_s);
-	if (ret != 0) {
-		IBTF_DPRINTF_L3(cmlog, "ibcm_arp_get_ibaddr: "
-		    "ibcm_arp_link_drivers failed %d", ret);
-		goto arp_ibaddr_error;
-	}
-
 	mutex_enter(&ib_s->lock);
 	ib_s->done = B_FALSE;
 	mutex_exit(&ib_s->lock);
 
-	ret = ibcm_arp_pr_lookup(ib_s, &destaddr, &srcaddr,
-	    ibcm_arp_get_ibaddr_cb);
+	ret = ibcm_resolver_pr_lookup(ib_s, &destaddr, &srcaddr);
 
-	IBTF_DPRINTF_L3(cmlog, "ibcm_arp_get_ibaddr: ibcm_arp_pr_lookup "
+	IBTF_DPRINTF_L3(cmlog, "ibcm_arp_get_ibaddr: ibcm_resolver_pr_lookup "
 	    "returned: %d", ret);
 	if (ret == 0) {
 		mutex_enter(&ib_s->lock);
@@ -393,7 +109,6 @@ ibcm_arp_get_ibaddr(ibt_ip_addr_t srcaddr, ibt_ip_addr_t destaddr,
 		mutex_exit(&ib_s->lock);
 	}
 
-	(void) ibcm_arp_unlink_drivers(ib_s);
 	mutex_enter(&ib_s->lock);
 	wqnp = ib_s->wqnp;
 	if (ib_s->status == 0) {
@@ -407,11 +122,11 @@ ibcm_arp_get_ibaddr(ibt_ip_addr_t srcaddr, ibt_ip_addr_t destaddr,
 		    ib_s->wqnp->sgid.gid_prefix, ib_s->wqnp->sgid.gid_guid,
 		    ib_s->wqnp->dgid.gid_prefix, ib_s->wqnp->dgid.gid_guid);
 
-		ibcm_arp_prwqn_delete(wqnp);
+		ibcm_arp_delete_prwqn(wqnp);
 	} else if (ret == 0) {
 		/*
 		 * We come here only when lookup has returned empty (failed)
-		 * via callback routine - ibcm_arp_get_ibaddr_cb
+		 * via callback routine.
 		 * i.e. ib_s->status is non-zero, while ret is zero.
 		 */
 		if (wqnp)
@@ -884,20 +599,3 @@ srcip_plist_end:
 
 	return (ret);
 }
-/* Routines for warlock */
-
-/* ARGSUSED */
-static int
-ibcm_arp_dummy_ibaddr_hdl(void *arg, int status)
-{
-	ibcm_arp_prwqn_t		dummy_wqn1;
-	ibcm_arp_prwqn_t		dummy_wqn2;
-
-	dummy_wqn1.func = ibcm_arp_get_ibaddr_cb;
-	dummy_wqn2.func = ibcm_arp_dummy_ibaddr_hdl;
-
-	IBTF_DPRINTF_L5(cmlog, "ibcm_arp_dummy_ibaddr_hdl: "
-	    "dummy_wqn1.func %p %p", dummy_wqn1.func, dummy_wqn2.func);
-
-	return (0);
-}
diff --git a/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp_link.c b/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp_link.c
index 79d420d467..45fbfd7932 100644
--- a/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp_link.c
+++ b/usr/src/uts/common/io/ib/mgt/ibcm/ibcm_arp_link.c
@@ -24,309 +24,32 @@
  */
 
 #include <sys/types.h>
-#include <sys/stream.h>
-#include <sys/dlpi.h>
-#include <sys/stropts.h>
-#include <sys/strsun.h>
-#include <sys/sysmacros.h>
-#include <sys/strlog.h>
-#include <sys/ddi.h>
-#include <sys/cmn_err.h>
-#include <sys/socket.h>
 #include <net/if.h>
 #include <net/if_types.h>
-#include <netinet/in.h>
-#include <sys/ethernet.h>
-#include <inet/arp.h>
 #include <inet/ip.h>
 #include <inet/ip_ire.h>
 #include <inet/ip_if.h>
 #include <sys/ib/mgt/ibcm/ibcm_arp.h>
-#include <inet/ip_ftable.h>
-
-static areq_t ibcm_arp_areq_template = {
-	AR_ENTRY_QUERY,	/* cmd */
-	sizeof (areq_t) + (2 * IP_ADDR_LEN),	/* name offset */
-	sizeof (areq_t),	/* name len */
-	IP_ARP_PROTO_TYPE,	/* protocol, from arps perspective */
-	sizeof (areq_t),	/* target addr offset */
-	IP_ADDR_LEN,	/* target ADDR_length */
-	0,	/* flags */
-	sizeof (areq_t) + IP_ADDR_LEN,	/* sender addr offset */
-	IP_ADDR_LEN,	/* sender addr length */
-	IBCM_ARP_XMIT_COUNT,	/* xmit_count */
-	IBCM_ARP_XMIT_INTERVAL,	/* (re)xmit_interval in milliseconds */
-	4	/* max # of requests to buffer */
-		/*
-		 * anything else filled in by the code
-		 */
-};
-
-static area_t ibcm_arp_area_template = {
-	AR_ENTRY_ADD,			/* cmd */
-	sizeof (area_t) + IPOIB_ADDRL + (2 * IP_ADDR_LEN), /* name offset */
-	sizeof (area_t),		/* name len */
-	IP_ARP_PROTO_TYPE,		/* protocol, from arps perspective */
-	sizeof (area_t),		/* proto addr offset */
-	IP_ADDR_LEN,			/* proto ADDR_length */
-	sizeof (area_t) + (IP_ADDR_LEN),	/* proto mask offset */
-	0,				/* flags */
-	sizeof (area_t) + (2 * IP_ADDR_LEN),	/* hw addr offset */
-	IPOIB_ADDRL				/* hw addr length */
-};
 
 extern char cmlog[];
 
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", msgb))
-_NOTE(SCHEME_PROTECTS_DATA("Unshared data", area_t))
 _NOTE(SCHEME_PROTECTS_DATA("Unshared data", ibcm_arp_streams_t))
 
-static void ibcm_arp_timeout(void *arg);
-static void ibcm_arp_pr_callback(ibcm_arp_prwqn_t *wqnp, int status);
-static void ibcm_ipv6_resolver_ack(ip2mac_t *, void *);
-static int ibcm_ipv6_lookup(ibcm_arp_prwqn_t *wqnp, ill_t *ill, zoneid_t zid);
-
-/*
- * issue a AR_ENTRY_QUERY to arp driver and schedule a timeout.
- */
-static int
-ibcm_arp_query_arp(ibcm_arp_prwqn_t *wqnp)
-{
-	int len;
-	int name_len;
-	int name_offset;
-	char *cp;
-	mblk_t *mp;
-	mblk_t *mp1;
-	areq_t *areqp;
-	ibcm_arp_streams_t *ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_query_arp(ib_s: %p wqnp: %p)",
-	    ib_s, wqnp);
-
-	name_offset = ibcm_arp_areq_template.areq_name_offset;
-
-	/*
-	 * allocate mblk for AR_ENTRY_QUERY
-	 */
-	name_len = strlen(wqnp->ifname) + 1;
-	len = name_len + name_offset;
-	if ((mp = allocb(len, BPRI_HI)) == NULL) {
-		return (ENOMEM);
-	}
-	bzero(mp->b_rptr, len);
-	mp->b_wptr += len;
-
-	/*
-	 * allocate a mblk and set wqnp in the data
-	 */
-	if ((mp1 = allocb(sizeof (void *), BPRI_HI)) == NULL) {
-		freeb(mp);
-		return (ENOMEM);
-	}
-
-	mp1->b_wptr += sizeof (void *);
-	*(uintptr_t *)(void *)mp1->b_rptr = (uintptr_t)wqnp;	/* store wqnp */
-
-	cp = (char *)mp->b_rptr;
-	bcopy(&ibcm_arp_areq_template, cp, sizeof (areq_t));
-	areqp = (void *)cp;
-	areqp->areq_name_length = name_len;
-
-	cp = (char *)areqp + areqp->areq_name_offset;
-	bcopy(wqnp->ifname, cp, name_len);
-
-	areqp->areq_proto = wqnp->ifproto;
-	bcopy(&wqnp->ifproto, areqp->areq_sap, 2);
-	cp = (char *)areqp + areqp->areq_target_addr_offset;
-	bcopy(&wqnp->dst_addr.un.ip4addr, cp, IP_ADDR_LEN);
-	cp = (char *)areqp + areqp->areq_sender_addr_offset;
-	bcopy(&wqnp->src_addr.un.ip4addr, cp, IP_ADDR_LEN);
-
-	mp->b_cont = mp1;
-
-	DB_TYPE(mp) = M_PROTO;
-
-	/*
-	 * issue the request to arp
-	 */
-	wqnp->flags |= IBCM_ARP_PR_RESOLVE_PENDING;
-	wqnp->timeout_id = timeout(ibcm_arp_timeout, wqnp,
-	    drv_usectohz(IBCM_ARP_TIMEOUT * 1000));
-	if (canputnext(ib_s->arpqueue)) {
-		putnext(ib_s->arpqueue, mp);
-	} else {
-		(void) putq(ib_s->arpqueue, mp);
-		qenable(ib_s->arpqueue);
-	}
-
-	return (0);
-}
-
-/*
- * issue AR_ENTRY_SQUERY to arp driver
- */
-static int
-ibcm_arp_squery_arp(ibcm_arp_prwqn_t *wqnp)
-{
-	int len;
-	int name_len;
-	char *cp;
-	mblk_t *mp;
-	mblk_t *mp1;
-	area_t *areap;
-	uint32_t  proto_mask = 0xffffffff;
-	struct iocblk *ioc;
-	ibcm_arp_streams_t *ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_squery_arp(ib_s: %p wqnp: %p)",
-	    ib_s, wqnp);
-
-	/*
-	 * allocate mblk for AR_ENTRY_SQUERY
-	 */
-	name_len = strlen(wqnp->ifname) + 1;
-	len = ibcm_arp_area_template.area_name_offset + name_len +
-	    sizeof (uintptr_t);
-	if ((mp = allocb(len, BPRI_HI)) == NULL) {
-		return (ENOMEM);
-	}
-	bzero(mp->b_rptr, len);
-	mp->b_wptr += len + sizeof (uintptr_t);
-
-	*(uintptr_t *)(void *)mp->b_rptr = (uintptr_t)wqnp;	/* store wqnp */
-	mp->b_rptr += sizeof (uintptr_t);
-
-
-	cp = (char *)mp->b_rptr;
-	bcopy(&ibcm_arp_area_template, cp, sizeof (area_t));
-
-	areap = (void *)cp;
-	areap->area_cmd = AR_ENTRY_SQUERY;
-	areap->area_name_length = name_len;
-	cp = (char *)areap + areap->area_name_offset;
-	bcopy(wqnp->ifname, cp, name_len);
-
-	cp = (char *)areap + areap->area_proto_addr_offset;
-	bcopy(&wqnp->dst_addr.un.ip4addr, cp, IP_ADDR_LEN);
-
-	cp = (char *)areap + areap->area_proto_mask_offset;
-	bcopy(&proto_mask, cp, IP_ADDR_LEN);
-
-	mp1 = allocb(sizeof (struct iocblk), BPRI_HI);
-	if (mp1 == NULL) {
-		freeb(mp);
-		return (ENOMEM);
-	}
-	ioc = (void *)mp1->b_rptr;
-	ioc->ioc_cmd = AR_ENTRY_SQUERY;
-	ioc->ioc_error = 0;
-	ioc->ioc_cr = NULL;
-	ioc->ioc_count = msgdsize(mp);
-	mp1->b_wptr += sizeof (struct iocblk);
-	mp1->b_cont = mp;
-
-	DB_TYPE(mp1) = M_IOCTL;
-
-	if (canputnext(ib_s->arpqueue)) {
-		putnext(ib_s->arpqueue, mp1);
-	} else {
-		(void) putq(ib_s->arpqueue, mp1);
-		qenable(ib_s->arpqueue);
-	}
-	return (0);
-}
-
-/*
- * issue a AR_ENTRY_ADD to arp driver
- * This is required as arp driver does not maintain a cache.
- */
-static int
-ibcm_arp_add(ibcm_arp_prwqn_t *wqnp)
-{
-	int len;
-	int name_len;
-	char *cp;
-	mblk_t *mp;
-	area_t *areap;
-	uint32_t  proto_mask = 0xffffffff;
-	ibcm_arp_streams_t *ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_add(ib_s: %p wqnp: %p)", ib_s, wqnp);
-
-	/*
-	 * allocate mblk for AR_ENTRY_ADD
-	 */
-
-	name_len = strlen(wqnp->ifname) + 1;
-	len = ibcm_arp_area_template.area_name_offset + name_len;
-	if ((mp = allocb(len, BPRI_HI)) == NULL) {
-		return (ENOMEM);
-	}
-	bzero(mp->b_rptr, len);
-	mp->b_wptr += len;
-
-	cp = (char *)mp->b_rptr;
-	bcopy(&ibcm_arp_area_template, cp, sizeof (area_t));
-
-	areap = (void *)mp->b_rptr;
-	areap->area_name_length = name_len;
-	cp = (char *)areap + areap->area_name_offset;
-	bcopy(wqnp->ifname, cp, name_len);
-
-	cp = (char *)areap + areap->area_proto_addr_offset;
-	bcopy(&wqnp->dst_addr.un.ip4addr, cp, IP_ADDR_LEN);
-
-	cp = (char *)areap + areap->area_proto_mask_offset;
-	bcopy(&proto_mask, cp, IP_ADDR_LEN);
-
-	cp = (char *)areap + areap->area_hw_addr_offset;
-	bcopy(&wqnp->dst_mac, cp, IPOIB_ADDRL);
-
-	DB_TYPE(mp) = M_PROTO;
-
-	if (canputnext(ib_s->arpqueue)) {
-		putnext(ib_s->arpqueue, mp);
-	} else {
-		(void) putq(ib_s->arpqueue, mp);
-		qenable(ib_s->arpqueue);
-	}
-	return (0);
-}
-
-
-/*
- * timeout routine when there is no response to AR_ENTRY_QUERY
- */
-static void
-ibcm_arp_timeout(void *arg)
-{
-	ibcm_arp_prwqn_t *wqnp = (ibcm_arp_prwqn_t *)arg;
-	ibcm_arp_streams_t *ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_timeout(ib_s: %p wqnp: %p)",
-	    ib_s, wqnp);
-	wqnp->flags &= ~IBCM_ARP_PR_RESOLVE_PENDING;
-	cv_broadcast(&ib_s->cv);
-
-	/*
-	 * indicate to user
-	 */
-	ibcm_arp_pr_callback(wqnp, EHOSTUNREACH);
-}
+static void ibcm_resolver_ack(ip2mac_t *, void *);
+static int ibcm_nce_lookup(ibcm_arp_prwqn_t *wqnp, ill_t *ill, zoneid_t zid);
 
 /*
  * delete a wait queue node from the list.
  * assumes mutex is acquired
  */
 void
-ibcm_arp_prwqn_delete(ibcm_arp_prwqn_t *wqnp)
+ibcm_arp_delete_prwqn(ibcm_arp_prwqn_t *wqnp)
 {
 	ibcm_arp_streams_t *ib_s;
 
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_prwqn_delete(%p)", wqnp);
+	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_delete_prwqn(%p)", wqnp);
 
-	ib_s = (ibcm_arp_streams_t *)wqnp->arg;
+	ib_s = wqnp->ib_str;
 	ib_s->wqnp = NULL;
 	kmem_free(wqnp, sizeof (ibcm_arp_prwqn_t));
 }
@@ -336,7 +59,7 @@ ibcm_arp_prwqn_delete(ibcm_arp_prwqn_t *wqnp)
  */
 static ibcm_arp_prwqn_t *
 ibcm_arp_create_prwqn(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
-    ibt_ip_addr_t *src_addr, ibcm_arp_pr_comp_func_t func)
+    ibt_ip_addr_t *src_addr)
 {
 	ibcm_arp_prwqn_t *wqnp;
 
@@ -354,8 +77,7 @@ ibcm_arp_create_prwqn(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	if (src_addr) {
 		wqnp->usrc_addr = *src_addr;
 	}
-	wqnp->func = func;
-	wqnp->arg = ib_s;
+	wqnp->ib_str = ib_s;
 	wqnp->ifproto = (dst_addr->family == AF_INET) ?
 	    ETHERTYPE_IP : ETHERTYPE_IPV6;
 
@@ -366,17 +88,6 @@ ibcm_arp_create_prwqn(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	return (wqnp);
 }
 
-/*
- * call the user function
- * called with lock held
- */
-static void
-ibcm_arp_pr_callback(ibcm_arp_prwqn_t *wqnp, int status)
-{
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_callback(%p, %d)", wqnp, status);
-
-	wqnp->func((void *)wqnp, status);
-}
 
 /*
  * Check if the interface is loopback or IB.
@@ -391,23 +102,24 @@ ibcm_arp_check_interface(ill_t *ill)
 }
 
 int
-ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
-    ibt_ip_addr_t *src_addr, ibcm_arp_pr_comp_func_t func)
+ibcm_resolver_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
+    ibt_ip_addr_t *src_addr)
 {
 	ibcm_arp_prwqn_t *wqnp;
 	ire_t	*ire = NULL;
-	ire_t	*src_ire = NULL;
-	ipif_t	*ipif;
-	ill_t	*ill, *hwaddr_ill = NULL;
+	ipif_t	*ipif = NULL;
+	ill_t	*ill = NULL;
+	ill_t	*hwaddr_ill = NULL;
 	ip_stack_t *ipst;
 	int		len;
+	ipaddr_t	setsrcv4;
+	in6_addr_t	setsrcv6;
 
 	IBCM_PRINT_IP("ibcm_arp_pr_lookup: SRC", src_addr);
 	IBCM_PRINT_IP("ibcm_arp_pr_lookup: DST", dst_addr);
 
-	if ((wqnp = ibcm_arp_create_prwqn(ib_s, dst_addr,
-	    src_addr, func)) == NULL) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
+	if ((wqnp = ibcm_arp_create_prwqn(ib_s, dst_addr, src_addr)) == NULL) {
+		IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
 		    "ibcm_arp_create_prwqn failed");
 		ib_s->status = ENOMEM;
 		return (1);
@@ -416,86 +128,111 @@ ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	ipst = netstack_find_by_zoneid(GLOBAL_ZONEID)->netstack_ip;
 	if (dst_addr->family == AF_INET) {
 		/*
-		 * Get the ire for the local address
+		 * A local address is always specified, and it is used
+		 * to find the zoneid.
 		 */
-		IBTF_DPRINTF_L5(cmlog, "ibcm_arp_pr_lookup: ire_ctable_lookup");
-		src_ire = ire_ctable_lookup(src_addr->un.ip4addr, NULL,
-		    IRE_LOCAL, NULL, ALL_ZONES, NULL, MATCH_IRE_TYPE, ipst);
-		if (src_ire == NULL) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ire_ctable_lookup failed");
+		ipif = ipif_lookup_addr(src_addr->un.ip4addr, NULL, ALL_ZONES,
+		    ipst);
+		if (ipif == NULL) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ipif_lookup_addr failed");
 			ib_s->status = EFAULT;
 			goto fail;
 		}
-		IBTF_DPRINTF_L5(cmlog, "ibcm_arp_pr_lookup: ire_ctable_lookup");
 
 		/*
-		 * get an ire for the destination address with the matching
-		 * source address
+		 * get an ire for the destination adress.
+		 * Note that we can't use MATCH_IRE_ILL since that would
+		 * require that the first ill we find have ire_ill set. Thus
+		 * we compare ire_ill against ipif_ill after the lookup.
 		 */
-		ire = ire_ftable_lookup(dst_addr->un.ip4addr, 0, 0, 0,
-		    src_ire->ire_ipif, 0, src_ire->ire_zoneid, 0, NULL,
-		    MATCH_IRE_SRC, ipst);
-		if (ire == NULL) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ire_ftable_lookup failed");
+		setsrcv4 = INADDR_ANY;
+		ire = ire_route_recursive_v4(dst_addr->un.ip4addr, 0, NULL,
+		    ipif->ipif_zoneid, NULL, MATCH_IRE_DSTONLY, B_TRUE, 0, ipst,
+		    &setsrcv4, NULL, NULL);
+
+		ASSERT(ire != NULL);
+		if (ire->ire_flags & (RTF_REJECT|RTF_BLACKHOLE)) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ire_route_recursive_v4 failed");
+			ib_s->status = EFAULT;
+			goto fail;
+		}
+		ill = ire_nexthop_ill(ire);
+		if (ill == NULL) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ire_nexthop_ill failed");
+			ib_s->status = EFAULT;
+			goto fail;
+		}
+		if (ill != ipif->ipif_ill) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "wrong ill");
 			ib_s->status = EFAULT;
 			goto fail;
 		}
 
-		IBTF_DPRINTF_L5(cmlog, "ibcm_arp_pr_lookup: ire_ftable_lookup:"
-		    "done");
-
-		wqnp->gateway.un.ip4addr =
-		    ((ire->ire_gateway_addr == INADDR_ANY) ?
-		    ire->ire_addr : ire->ire_gateway_addr);
+		wqnp->gateway.un.ip4addr = ire->ire_gateway_addr;
 		wqnp->netmask.un.ip4addr = ire->ire_mask;
-		wqnp->src_addr.un.ip4addr = ire->ire_src_addr;
+		wqnp->src_addr.un.ip4addr = src_addr->un.ip4addr;
 		wqnp->src_addr.family = wqnp->gateway.family =
 		    wqnp->netmask.family = AF_INET;
 
 	} else if (dst_addr->family == AF_INET6) {
 		/*
-		 * Get the ire for the local address
+		 * A local address is always specified, and it is used
+		 * to find the zoneid.
+		 * We should really match on scopeid for link locals here.
 		 */
-		src_ire = ire_ctable_lookup_v6(&src_addr->un.ip6addr, NULL,
-		    IRE_LOCAL, NULL, ALL_ZONES, NULL, MATCH_IRE_TYPE, ipst);
-		if (src_ire == NULL) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ire_ctable_lookup_v6 failed");
+		ipif = ipif_lookup_addr_v6(&src_addr->un.ip6addr, NULL,
+		    ALL_ZONES, ipst);
+		if (ipif == NULL) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ipif_lookup_addr_v6 failed");
 			ib_s->status = EFAULT;
 			goto fail;
 		}
-		IBTF_DPRINTF_L5(cmlog, "ibcm_arp_pr_lookup: "
-		    "ire_ctable_lookup_v6: done");
 
 		/*
-		 * get an ire for the destination address with the matching
-		 * source address
+		 * get an ire for the destination adress.
+		 * Note that we can't use MATCH_IRE_ILL since that would
+		 * require that the first ill we find have ire_ill set. Thus
+		 * we compare ire_ill against ipif_ill after the lookup.
 		 */
-		ire = ire_ftable_lookup_v6(&dst_addr->un.ip6addr, 0, 0, 0,
-		    src_ire->ire_ipif, 0, src_ire->ire_zoneid, 0, NULL,
-		    MATCH_IRE_SRC, ipst);
-		if (ire == NULL) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ire_ftable_lookup_v6 failed");
+		setsrcv6 = ipv6_all_zeros;
+		ire = ire_route_recursive_v6(&dst_addr->un.ip6addr, 0, NULL,
+		    ipif->ipif_zoneid, NULL, MATCH_IRE_DSTONLY, B_TRUE, 0, ipst,
+		    &setsrcv6, NULL, NULL);
+
+		ASSERT(ire != NULL);
+		if (ire->ire_flags & (RTF_REJECT|RTF_BLACKHOLE)) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ire_route_recursive_v6 failed");
+			ib_s->status = EFAULT;
+			goto fail;
+		}
+		ill = ire_nexthop_ill(ire);
+		if (ill == NULL) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "ire_nexthop_ill failed");
+			ib_s->status = EFAULT;
+			goto fail;
+		}
+
+		if (ill != ipif->ipif_ill) {
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "wrong ill");
 			ib_s->status = EFAULT;
 			goto fail;
 		}
-		IBTF_DPRINTF_L5(cmlog, "ibcm_arp_pr_lookup: "
-		    "ire_ftable_lookup_v6: done");
 
-		wqnp->gateway.un.ip6addr =
-		    (IN6_IS_ADDR_UNSPECIFIED(&ire->ire_gateway_addr_v6) ?
-		    ire->ire_addr_v6 : ire->ire_gateway_addr_v6);
+		wqnp->gateway.un.ip6addr = ire->ire_gateway_addr_v6;
 		wqnp->netmask.un.ip6addr = ire->ire_mask_v6;
-		wqnp->src_addr.un.ip6addr = ire->ire_src_addr_v6;
+		wqnp->src_addr.un.ip6addr = src_addr->un.ip6addr;
 		wqnp->src_addr.family = wqnp->gateway.family =
 		    wqnp->netmask.family = AF_INET6;
 	}
 
-	ipif = src_ire->ire_ipif;
-	ill = ipif->ipif_ill;
 	(void) strlcpy(wqnp->ifname, ill->ill_name, sizeof (wqnp->ifname));
 
 	/*
@@ -504,18 +241,19 @@ ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	 */
 	if (IS_IPMP(ill)) {
 		if ((hwaddr_ill = ipmp_ipif_hold_bound_ill(ipif)) == NULL) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: no bound "
-			    "ill for IPMP interface %s", ill->ill_name);
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "no bound ill for IPMP interface %s",
+			    ill->ill_name);
 			ib_s->status = EFAULT;
 			goto fail;
 		}
 	} else {
 		hwaddr_ill = ill;
-		ill_refhold(hwaddr_ill); 	/* for symmetry */
+		ill_refhold(hwaddr_ill);	/* for symmetry */
 	}
 
 	if ((ib_s->status = ibcm_arp_check_interface(hwaddr_ill)) != 0) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
+		IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
 		    "ibcm_arp_check_interface failed");
 		goto fail;
 	}
@@ -523,7 +261,7 @@ ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	bcopy(hwaddr_ill->ill_phys_addr, &wqnp->src_mac,
 	    hwaddr_ill->ill_phys_addr_length);
 
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_lookup: outgoing if:%s",
+	IBTF_DPRINTF_L4(cmlog, "ibcm_resolver_pr_lookup: outgoing if:%s",
 	    wqnp->ifname);
 
 	/*
@@ -534,8 +272,8 @@ ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 		len = (wqnp->usrc_addr.family == AF_INET) ?
 		    IP_ADDR_LEN : sizeof (in6_addr_t);
 		if (bcmp(&wqnp->usrc_addr.un, &wqnp->src_addr.un, len)) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: srcaddr "
-			    "mismatch:%d", ENETUNREACH);
+			IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+			    "srcaddr mismatch:%d", ENETUNREACH);
 			goto fail;
 		}
 	}
@@ -545,253 +283,77 @@ ibcm_arp_pr_lookup(ibcm_arp_streams_t *ib_s, ibt_ip_addr_t *dst_addr,
 	 * interface, now get the destination mac address from
 	 * arp or ipv6 drivers
 	 */
-	if (wqnp->dst_addr.family == AF_INET) {
-		if ((ib_s->status = ibcm_arp_squery_arp(wqnp)) != 0) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ibcm_arp_squery_arp failed: %d", ib_s->status);
-			goto fail;
-		}
-	} else {
-		if ((ib_s->status = ibcm_ipv6_lookup(wqnp, ill, getzoneid())) !=
-		    0) {
-			IBTF_DPRINTF_L2(cmlog, "ibcm_arp_pr_lookup: "
-			    "ibcm_ipv6_lookup failed: %d", ib_s->status);
-			goto fail;
-		}
+	ib_s->status = ibcm_nce_lookup(wqnp, ill, getzoneid());
+	if (ib_s->status != 0) {
+		IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_pr_lookup: "
+		    "ibcm_nce_lookup failed: %d", ib_s->status);
+		goto fail;
 	}
 
 	ill_refrele(hwaddr_ill);
-	IRE_REFRELE(ire);
-	IRE_REFRELE(src_ire);
+	ill_refrele(ill);
+	ire_refrele(ire);
+	ipif_refrele(ipif);
 	netstack_rele(ipst->ips_netstack);
 
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_lookup: Return: 0x%p", wqnp);
+	IBTF_DPRINTF_L4(cmlog, "ibcm_resolver_pr_lookup: Return: 0x%p", wqnp);
 	return (0);
 fail:
 	if (hwaddr_ill != NULL)
 		ill_refrele(hwaddr_ill);
+	if (ill != NULL)
+		ill_refrele(ill);
 	if (ire != NULL)
-		IRE_REFRELE(ire);
-	if (src_ire != NULL)
-		IRE_REFRELE(src_ire);
-	ibcm_arp_prwqn_delete(wqnp);
+		ire_refrele(ire);
+	if (ipif != NULL)
+		ipif_refrele(ipif);
+	ibcm_arp_delete_prwqn(wqnp);
 	netstack_rele(ipst->ips_netstack);
 	return (1);
 }
 
 /*
- * called from lrsrv.
- * process a AR_ENTRY_QUERY reply from arp
- * the message should be M_DATA -->> dl_unitdata_req
- */
-static void
-ibcm_arp_pr_arp_query_ack(mblk_t *mp)
-{
-	ibcm_arp_prwqn_t 	*wqnp;
-	dl_unitdata_req_t *dlreq;
-	ibcm_arp_streams_t *ib_s;
-	char *cp;
-	int rc;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_arp_query_ack(%p)", mp);
-
-	/*
-	 * the first mblk contains the wqnp pointer for the request
-	 */
-	if (MBLKL(mp) != sizeof (void *)) {
-		freemsg(mp);
-		return;
-	}
-
-	wqnp = *(ibcm_arp_prwqn_t **)(void *)mp->b_rptr; /* retrieve wqnp */
-	ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	mutex_enter(&ib_s->lock);
-
-	/*
-	 * cancel the timeout for this request
-	 */
-	(void) untimeout(wqnp->timeout_id);
-
-	/*
-	 * sanity checks on the dl_unitdata_req block
-	 */
-	if (!mp->b_cont) {
-		IBTF_DPRINTF_L2(cmlog, "areq_ack: b_cont = NULL\n");
-		rc = EPROTO;
-		goto user_callback;
-	}
-	if (MBLKL(mp->b_cont) < (sizeof (dl_unitdata_req_t) + IPOIB_ADDRL)) {
-		IBTF_DPRINTF_L2(cmlog, "areq_ack: invalid len in "
-		    "dl_unitdatareq_t block\n");
-		rc = EPROTO;
-		goto user_callback;
-	}
-	dlreq = (void *)mp->b_cont->b_rptr;
-	if (dlreq->dl_primitive != DL_UNITDATA_REQ) {
-		IBTF_DPRINTF_L2(cmlog, "areq_ack: invalid dl_primitive "
-		    "in dl_unitdatareq_t block\n");
-		rc = EPROTO;
-		goto user_callback;
-	}
-	if (dlreq->dl_dest_addr_length != (IPOIB_ADDRL + 2)) {
-		IBTF_DPRINTF_L2(cmlog, "areq_ack: invalid hw len in "
-		    "dl_unitdatareq_t block %d\n", dlreq->dl_dest_addr_length);
-		rc = EPROTO;
-		goto user_callback;
-	}
-	cp = (char *)mp->b_cont->b_rptr + dlreq->dl_dest_addr_offset;
-	bcopy(cp, &wqnp->dst_mac, IPOIB_ADDRL);
-
-	/*
-	 * at this point we have src/dst gid's derived from the mac addresses
-	 * now get the hca, port
-	 */
-	bcopy(&wqnp->src_mac.ipoib_gidpref, &wqnp->sgid, sizeof (ib_gid_t));
-	bcopy(&wqnp->dst_mac.ipoib_gidpref, &wqnp->dgid, sizeof (ib_gid_t));
-	freemsg(mp);
-
-	IBCM_H2N_GID(wqnp->sgid);
-	IBCM_H2N_GID(wqnp->dgid);
-
-	(void) ibcm_arp_add(wqnp);
-
-	mutex_exit(&ib_s->lock);
-	ibcm_arp_pr_callback(wqnp, 0);
-
-	return;
-user_callback:
-	freemsg(mp);
-	mutex_exit(&ib_s->lock);
-
-	/*
-	 * indicate to user
-	 */
-	ibcm_arp_pr_callback(wqnp, rc);
-}
-
-/*
- * process a AR_ENTRY_SQUERY reply from arp
- * the message should be M_IOCACK -->> area_t
+ * Query the neighbor cache for IPv4/IPv6 to mac address mapping.
  */
-static void
-ibcm_arp_pr_arp_squery_ack(mblk_t *mp)
+static int
+ibcm_nce_lookup(ibcm_arp_prwqn_t *wqnp, ill_t *ill, zoneid_t zoneid)
 {
-	struct iocblk *ioc;
-	mblk_t	*mp1;
-	ibcm_arp_prwqn_t 	*wqnp;
-	ibcm_arp_streams_t *ib_s;
-	area_t *areap;
-	char *cp;
-
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_arp_squery_ack(%p)", mp);
-
-	if (MBLKL(mp) < sizeof (struct iocblk)) {
-		freemsg(mp);
-		return;
-	}
-
-	ioc = (void *)mp->b_rptr;
-	if ((ioc->ioc_cmd != AR_ENTRY_SQUERY) || (mp->b_cont == NULL)) {
-		freemsg(mp);
-		return;
-	}
-
-	mp1 = mp->b_cont;
-
-	wqnp = *(ibcm_arp_prwqn_t **)((uintptr_t)mp1->b_rptr -
-	    sizeof (uintptr_t));
-	ib_s = (ibcm_arp_streams_t *)wqnp->arg;
-
-	mutex_enter(&ib_s->lock);
-
-	/*
-	 * cancel the timeout for this request
-	 */
-	(void) untimeout(wqnp->timeout_id);
-
-	/* If the entry was not in arp cache, ioc_error is set */
-	if (ioc->ioc_error) {
-
-		/*
-		 * send out AR_ENTRY_QUERY which would send
-		 * arp-request on wire
-		 */
-		IBTF_DPRINTF_L3(cmlog, "Sending a Query_ARP");
-
-		(void) ibcm_arp_query_arp(wqnp);
-		freemsg(mp);
-		mutex_exit(&ib_s->lock);
-		return;
+	ip2mac_t	ip2m;
+	sin_t		*sin;
+	sin6_t		*sin6;
+	ip2mac_id_t	ip2mid;
+	int		err;
+
+	if (wqnp->src_addr.family != wqnp->dst_addr.family) {
+		IBTF_DPRINTF_L2(cmlog, "ibcm_nce_lookup: Mis-match SRC_ADDR "
+		    "Family: %d, DST_ADDR Family %d", wqnp->src_addr.family,
+		    wqnp->dst_addr.family);
+		return (1);
 	}
+	bzero(&ip2m, sizeof (ip2m));
 
-	areap = (void *)mp1->b_rptr;
-	cp = (char *)areap + areap->area_hw_addr_offset;
-	bcopy(cp, &wqnp->dst_mac, IPOIB_ADDRL);
-
-	/*
-	 * at this point we have src/dst gid's derived from the mac addresses
-	 * now get the hca, port
-	 */
-	bcopy(&wqnp->src_mac.ipoib_gidpref, &wqnp->sgid, sizeof (ib_gid_t));
-	bcopy(&wqnp->dst_mac.ipoib_gidpref, &wqnp->dgid, sizeof (ib_gid_t));
-	freemsg(mp);
-
-	IBCM_H2N_GID(wqnp->sgid);
-	IBCM_H2N_GID(wqnp->dgid);
-
-	mutex_exit(&ib_s->lock);
-	ibcm_arp_pr_callback(wqnp, 0);
-}
-
-/*
- * Process arp ack's.
- */
-void
-ibcm_arp_pr_arp_ack(mblk_t *mp)
-{
-	IBTF_DPRINTF_L4(cmlog, "ibcm_arp_pr_arp_ack(0x%p, DB_TYPE %lX)",
-	    mp, DB_TYPE(mp));
-
-	if (DB_TYPE(mp) == M_DATA) {
-		ibcm_arp_pr_arp_query_ack(mp);
-	} else if ((DB_TYPE(mp) == M_IOCACK) ||
-	    (DB_TYPE(mp) == M_IOCNAK)) {
-		ibcm_arp_pr_arp_squery_ack(mp);
+	if (wqnp->dst_addr.family == AF_INET) {
+		sin = (sin_t *)&ip2m.ip2mac_pa;
+		sin->sin_family = AF_INET;
+		sin->sin_addr.s_addr = wqnp->dst_addr.un.ip4addr;
+	} else if (wqnp->dst_addr.family == AF_INET6) {
+		sin6 = (sin6_t *)&ip2m.ip2mac_pa;
+		sin6->sin6_family = AF_INET6;
+		sin6->sin6_addr = wqnp->dst_addr.un.ip6addr;
 	} else {
-		freemsg(mp);
-	}
-}
-
-/*
- * query the ipv6 driver cache for ipv6 to mac address mapping.
- */
-static int
-ibcm_ipv6_lookup(ibcm_arp_prwqn_t *wqnp, ill_t *ill, zoneid_t zoneid)
-{
-	ip2mac_t ip2m;
-	sin6_t *sin6;
-	ip2mac_id_t ip2mid;
-	int err;
-
-	if (wqnp->src_addr.family != AF_INET6) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_ipv6_lookup: SRC_ADDR NOT INET6: "
-		    "%d", wqnp->src_addr.family);
+		IBTF_DPRINTF_L2(cmlog, "ibcm_nce_lookup: Invalid DST_ADDR "
+		    "Family: %d", wqnp->dst_addr.family);
 		return (1);
 	}
 
-	bzero(&ip2m, sizeof (ip2m));
-	sin6 = (sin6_t *)&ip2m.ip2mac_pa;
-	sin6->sin6_family = AF_INET6;
-	sin6->sin6_addr = wqnp->dst_addr.un.ip6addr;
 	ip2m.ip2mac_ifindex = ill->ill_phyint->phyint_ifindex;
 
 	wqnp->flags |= IBCM_ARP_PR_RESOLVE_PENDING;
+
 	/*
-	 * XXX XTBD set the scopeid?
 	 * issue the request to IP for Neighbor Discovery
 	 */
-	ip2mid = ip2mac(IP2MAC_RESOLVE, &ip2m, ibcm_ipv6_resolver_ack, wqnp,
+	ip2mid = ip2mac(IP2MAC_RESOLVE, &ip2m, ibcm_resolver_ack, wqnp,
 	    zoneid);
 	err = ip2m.ip2mac_err;
 	if (err == EINPROGRESS) {
@@ -799,7 +361,7 @@ ibcm_ipv6_lookup(ibcm_arp_prwqn_t *wqnp, ill_t *ill, zoneid_t zoneid)
 		wqnp->flags |= IBCM_ARP_PR_RESOLVE_PENDING;
 		err = 0;
 	} else if (err == 0) {
-		ibcm_ipv6_resolver_ack(&ip2m, wqnp);
+		ibcm_resolver_ack(&ip2m, wqnp);
 	}
 	return (err);
 }
@@ -822,16 +384,16 @@ ibcm_check_sockdl(struct sockaddr_dl *sdl)
  * If Address resolution was succesful: return GID info.
  */
 static void
-ibcm_ipv6_resolver_ack(ip2mac_t *ip2macp, void *arg)
+ibcm_resolver_ack(ip2mac_t *ip2macp, void *arg)
 {
 	ibcm_arp_prwqn_t *wqnp = (ibcm_arp_prwqn_t *)arg;
 	ibcm_arp_streams_t *ib_s;
 	uchar_t *cp;
 	int err = 0;
 
-	IBTF_DPRINTF_L4(cmlog, "ibcm_ipv6_resolver_ack(%p, %p)", ip2macp, wqnp);
+	IBTF_DPRINTF_L4(cmlog, "ibcm_resolver_ack(%p, %p)", ip2macp, wqnp);
 
-	ib_s = (ibcm_arp_streams_t *)wqnp->arg;
+	ib_s = wqnp->ib_str;
 	mutex_enter(&ib_s->lock);
 
 	if (ip2macp->ip2mac_err != 0) {
@@ -842,7 +404,7 @@ ibcm_ipv6_resolver_ack(ip2mac_t *ip2macp, void *arg)
 	}
 
 	if (!ibcm_check_sockdl(&ip2macp->ip2mac_ha)) {
-		IBTF_DPRINTF_L2(cmlog, "ibcm_ipv6_resolver_ack: Error: "
+		IBTF_DPRINTF_L2(cmlog, "ibcm_resolver_ack: Error: "
 		    "interface %s is not IB\n", wqnp->ifname);
 		err = EHOSTUNREACH;
 		goto user_callback;
@@ -862,6 +424,11 @@ ibcm_ipv6_resolver_ack(ip2mac_t *ip2macp, void *arg)
 	IBCM_H2N_GID(wqnp->dgid);
 
 user_callback:
+
+	ib_s->status = err;
+	ib_s->done = B_TRUE;
+
+	/* lock is held by the caller. */
+	cv_signal(&ib_s->cv);
 	mutex_exit(&ib_s->lock);
-	ibcm_arp_pr_callback(wqnp, err);
 }
diff --git a/usr/src/uts/common/io/mac/mac_util.c b/usr/src/uts/common/io/mac/mac_util.c
index 0d342fdd93..88468b353e 100644
--- a/usr/src/uts/common/io/mac/mac_util.c
+++ b/usr/src/uts/common/io/mac/mac_util.c
@@ -476,7 +476,7 @@ mac_ip_hdr_length_v6(mblk_t *mp, ip6_t *ip6h, uint16_t *hdr_length,
 	endptr = mp->b_wptr;
 	if (((uchar_t *)ip6h + IPV6_HDR_LEN) > endptr)
 		return (B_FALSE);
-	ASSERT((IPH_HDR_VERSION(ip6h) & ~IP_FORWARD_PROG_BIT) == IPV6_VERSION);
+	ASSERT(IPH_HDR_VERSION(ip6h) == IPV6_VERSION);
 	length = IPV6_HDR_LEN;
 	whereptr = ((uint8_t *)&ip6h[1]); /* point to next hdr */
 
diff --git a/usr/src/uts/common/io/softmac/softmac_dev.c b/usr/src/uts/common/io/softmac/softmac_dev.c
index 23f43ced0b..eeb09fcb0b 100644
--- a/usr/src/uts/common/io/softmac/softmac_dev.c
+++ b/usr/src/uts/common/io/softmac/softmac_dev.c
@@ -146,6 +146,9 @@ static struct modlinkage softmac_modlinkage = {
 	NULL
 };
 
+static void softmac_dedicated_rx(void *, mac_resource_handle_t, mblk_t *,
+    mac_header_info_t *);
+
 /*ARGSUSED*/
 static int
 softmac_upper_constructor(void *buf, void *arg, int kmflag)
@@ -367,7 +370,8 @@ softmac_mod_rput(queue_t *rq, mblk_t *mp)
 		if (dlp->dl_primitive == DL_UNITDATA_IND) {
 
 			if ((rxinfo = slp->sl_rxinfo) != NULL) {
-				rxinfo->slr_rx(rxinfo->slr_arg, NULL, mp, NULL);
+				softmac_dedicated_rx(slp->sl_sup, NULL, mp,
+				    NULL);
 				break;
 			}
 
diff --git a/usr/src/uts/common/io/softmac/softmac_fp.c b/usr/src/uts/common/io/softmac/softmac_fp.c
index 7a10aa68b7..2fc66e9bd3 100644
--- a/usr/src/uts/common/io/softmac/softmac_fp.c
+++ b/usr/src/uts/common/io/softmac/softmac_fp.c
@@ -674,9 +674,12 @@ softmac_wput_single_nondata(softmac_upper_t *sup, mblk_t *mp)
 	t_uscalar_t	prim;
 
 	dbtype = DB_TYPE(mp);
+	sup->su_is_arp = 0;
 	switch (dbtype) {
-	case M_IOCTL:
-	case M_CTL: {
+	case M_CTL:
+		sup->su_is_arp = 1;
+		/* FALLTHROUGH */
+	case M_IOCTL: {
 		uint32_t	expected_mode;
 
 		if (((struct iocblk *)(mp->b_rptr))->ioc_cmd != SIOCSLIFNAME)
@@ -1132,7 +1135,10 @@ softmac_datapath_switch(softmac_t *softmac, boolean_t disable, boolean_t admin)
 			break;
 
 		req->ssq_expected_mode = expected_mode;
-
+		if (sup->su_is_arp) {
+			list_insert_tail(&reqlist, req);
+			continue;
+		}
 		/*
 		 * Allocate the DL_NOTE_REPLUMB message.
 		 */
@@ -1174,18 +1180,19 @@ softmac_datapath_switch(softmac_t *softmac, boolean_t disable, boolean_t admin)
 	 */
 	for (sup = list_head(&softmac->smac_sup_list); sup != NULL;
 	    sup = list_next(&softmac->smac_sup_list, sup)) {
-		mp = head->b_next;
-		head->b_next = NULL;
-
+		if (!sup->su_is_arp) {
+			mp = head->b_next;
+			head->b_next = NULL;
+			softmac_wput_nondata(sup, head);
+			head = mp;
+		}
 		/*
-		 * Add the swtich request to the requests list of the stream.
+		 * Add the switch request to the requests list of the stream.
 		 */
 		req = list_head(&reqlist);
 		ASSERT(req != NULL);
 		list_remove(&reqlist, req);
 		list_insert_tail(&sup->su_req_list, req);
-		softmac_wput_nondata(sup, head);
-		head = mp;
 	}
 
 	mutex_exit(&softmac->smac_fp_mutex);
diff --git a/usr/src/uts/common/io/stream.c b/usr/src/uts/common/io/stream.c
index b23036e9c5..658735b784 100644
--- a/usr/src/uts/common/io/stream.c
+++ b/usr/src/uts/common/io/stream.c
@@ -1605,7 +1605,9 @@ pullupmsg(mblk_t *mp, ssize_t len)
 		ASSERT(bp->b_datap->db_ref > 0);
 		ASSERT(bp->b_wptr >= bp->b_rptr);
 		n = MIN(bp->b_wptr - bp->b_rptr, len);
-		bcopy(bp->b_rptr, mp->b_wptr, (size_t)n);
+		ASSERT(n >= 0);		/* allow zero-length mblk_t's */
+		if (n > 0)
+			bcopy(bp->b_rptr, mp->b_wptr, (size_t)n);
 		mp->b_wptr += n;
 		bp->b_rptr += n;
 		len -= n;
diff --git a/usr/src/uts/common/io/strplumb.c b/usr/src/uts/common/io/strplumb.c
index f43648fd7f..473f7bc72e 100644
--- a/usr/src/uts/common/io/strplumb.c
+++ b/usr/src/uts/common/io/strplumb.c
@@ -53,17 +53,6 @@
 #include	<sys/esunddi.h>
 #include	<sys/promif.h>
 
-#include	<netinet/in.h>
-#include	<netinet/ip6.h>
-#include	<netinet/icmp6.h>
-#include	<netinet/sctp.h>
-#include	<inet/common.h>
-#include	<inet/ip.h>
-#include	<inet/ip6.h>
-#include	<inet/tcp.h>
-#include	<inet/sctp_ip.h>
-#include	<inet/udp_impl.h>
-
 #include	<sys/strlog.h>
 #include	<sys/log.h>
 #include	<sys/ethernet.h>
@@ -222,104 +211,6 @@ strplumb_init(void)
 	return (0);
 }
 
-static int
-strplumb_autopush(void)
-{
-	major_t		maj;
-	minor_t		min;
-	char		*mods[5];
-	uint_t		anchor = 1;
-	int		err;
-
-	min = (minor_t)-1;
-	mods[1] = NULL;
-
-	/*
-	 * ARP
-	 */
-	DBG0("setting up arp autopush\n");
-
-	mods[0] = ARP;
-
-	maj = ddi_name_to_major(ARP);
-	if ((err = kstr_autopush(SET_AUTOPUSH, &maj, &min, NULL, &anchor,
-	    mods)) != 0) {
-		printf("strplumb: kstr_autopush(SET/ARP) failed: %d\n", err);
-		return (err);
-	}
-
-	return (0);
-}
-
-static int
-strplumb_sctpq(ldi_ident_t li)
-{
-	ldi_handle_t	lh = NULL;
-	int		err;
-	int		rval;
-
-	DBG0("configuring SCTP default queue\n");
-
-	if ((err = ldi_open_by_name(SCTP6DEV, FREAD|FWRITE, CRED(), &lh,
-	    li)) != 0) {
-		printf("strplumb: open of SCTP6DEV failed: %d\n", err);
-		return (err);
-	}
-
-	if ((err = ldi_ioctl(lh, SCTP_IOC_DEFAULT_Q, (intptr_t)0, FKIOCTL,
-	    CRED(), &rval)) != 0) {
-		printf("strplumb: failed to set SCTP default queue: %d\n",
-		    err);
-		(void) ldi_close(lh, FREAD|FWRITE, CRED());
-		return (err);
-	}
-
-	return (0);
-}
-
-static int
-strplumb_tcpq(ldi_ident_t li)
-{
-	ldi_handle_t	lh = NULL;
-	ldi_handle_t	ip_lh = NULL;
-	int		err;
-	int		rval;
-
-	DBG0("configuring TCP default queue\n");
-
-	/*
-	 * We open IP6DEV here because we need to have it open to in
-	 * order to open TCP6DEV successfully.
-	 */
-	if ((err = ldi_open_by_name(IP6DEV, FREAD|FWRITE, CRED(), &ip_lh,
-	    li)) != 0) {
-		printf("strplumb: open of IP6DEV failed: %d\n", err);
-		return (err);
-	}
-
-	/*
-	 * We set the tcp default queue to IPv6 because IPv4 falls back to
-	 * IPv6 when it can't find a client, but IPv6 does not fall back to
-	 * IPv4.
-	 */
-	if ((err = ldi_open_by_name(TCP6DEV, FREAD|FWRITE, CRED(), &lh,
-	    li)) != 0) {
-		printf("strplumb: open of TCP6DEV failed: %d\n", err);
-		goto done;
-	}
-
-	if ((err = ldi_ioctl(lh, TCP_IOC_DEFAULT_Q, (intptr_t)0, FKIOCTL,
-	    CRED(), &rval)) != 0) {
-		printf("strplumb: failed to set TCP default queue: %d\n",
-		    err);
-		goto done;
-	}
-
-done:
-	(void) ldi_close(ip_lh, FREAD|FWRITE, CRED());
-	return (err);
-}
-
 /*
  * Can be set in /etc/system in the case of local booting. See comment below.
  */
@@ -447,11 +338,8 @@ strplumb_dev(ldi_ident_t li)
 
 	/*
 	 * Now set up the links. Ultimately, we should have two streams
-	 * permanently linked underneath UDP (which is actually IP with UDP
-	 * autopushed). One stream consists of the ARP-[ifname] combination,
-	 * while the other consists of ARP-IP-[ifname]. The second combination
-	 * seems a little weird, but is linked underneath UDP just to keep it
-	 * around.
+	 * permanently linked under UDP.  One stream consists of the
+	 * ARP-[ifname] combination, while the other consists of IP-[ifname].
 	 *
 	 * We pin underneath UDP here to match what is done in ifconfig(1m);
 	 * otherwise, ifconfig will be unable to unplumb the stream (the major
@@ -462,7 +350,7 @@ strplumb_dev(ldi_ident_t li)
 	 */
 
 	/*
-	 * Plumb UDP-ARP-IP-<dev>
+	 * Plumb UDP-IP-<dev>
 	 */
 
 	if ((err = ldi_open_by_name(rootfs.bo_devname, FREAD|FWRITE, CRED(),
@@ -494,12 +382,6 @@ strplumb_dev(ldi_ident_t li)
 		lifr.lifr_flags &= ~IFF_IPV4;
 		name = UDP6DEV;
 	}
-	if ((err = ldi_ioctl(lh, I_PUSH, (intptr_t)ARP, FKIOCTL, CRED(),
-	    &rval)) != 0) {
-		printf("strplumb: push ARP failed: %d\n", err);
-		goto done;
-	}
-
 	(void) strlcpy(lifr.lifr_name, rootfs.bo_ifname,
 	    sizeof (lifr.lifr_name));
 	lifr.lifr_ppa = rootfs.bo_ppa;
@@ -507,29 +389,17 @@ strplumb_dev(ldi_ident_t li)
 	if ((err = setifname(lh, &lifr)) != 0)
 		goto done;
 
-	/* Get the flags and check if ARP is needed */
+	/* get the flags and check if ARP is needed */
 	if ((err = getifflags(lh, &lifr)) != 0) {
 		printf("strplumb: getifflags %s IP failed, error %d\n",
 		    lifr.lifr_name, err);
 		goto done;
 	}
-
-	/* Pop out ARP if not needed */
-	if (lifr.lifr_flags & (IFF_NOARP | IFF_IPV6)) {
-		err = ldi_ioctl(lh, I_POP, (intptr_t)0, FKIOCTL, CRED(),
-		    &rval);
-		if (err != 0) {
-			printf("strplumb: pop ARP failed, error %d\n", err);
-			goto done;
-		}
-	}
-
 	if ((err = ldi_open_by_name(name, FREAD|FWRITE, CRED(), &mux_lh,
 	    li)) != 0) {
 		printf("strplumb: open of %s failed: %d\n", name, err);
 		goto done;
 	}
-
 	if ((err = ldi_ioctl(mux_lh, I_PLINK, (intptr_t)lh,
 	    FREAD|FWRITE|FNOCTTY|FKIOCTL, CRED(),
 	    &(ifr.ifr_ip_muxid))) != 0) {
@@ -538,9 +408,9 @@ strplumb_dev(ldi_ident_t li)
 		goto done;
 	}
 
-	if (af == AF_INET6) {
+	/* if ARP is not needed, we are done */
+	if (lifr.lifr_flags & (IFF_NOARP | IFF_IPV6))
 		goto done;
-	}
 
 	DBG2("UDP-ARP-IP-%s muxid: %d\n", rootfs.bo_ifname, ifr.ifr_ip_muxid);
 
@@ -610,22 +480,9 @@ strplumb(void)
 	if ((err = strplumb_init()) != 0)
 		return (err);
 
-	if ((err = strplumb_autopush()) != 0)
-		return (err);
-
 	if ((err = ldi_ident_from_mod(&modlinkage, &li)) != 0)
 		return (err);
 
-	/*
-	 * Setup the TCP and SCTP default queues for the global stack.
-	 * tcp/sctp_stack_init will do this for additional stack instances.
-	 */
-	if ((err = strplumb_sctpq(li)) != 0)
-		goto done;
-
-	if ((err = strplumb_tcpq(li)) != 0)
-		goto done;
-
 	if ((err = resolve_boot_path()) != 0)
 		goto done;
 
diff --git a/usr/src/uts/common/io/tl.c b/usr/src/uts/common/io/tl.c
index 7ddb24cddb..83f8cf6944 100644
--- a/usr/src/uts/common/io/tl.c
+++ b/usr/src/uts/common/io/tl.c
@@ -452,7 +452,7 @@ opdes_t	tl_opt_arr[] = {
 		OA_R,
 		OA_R,
 		OP_NP,
-		OP_PASSNEXT,
+		0,
 		sizeof (t_scalar_t),
 		0
 	},
@@ -462,7 +462,7 @@ opdes_t	tl_opt_arr[] = {
 		OA_RW,
 		OA_RW,
 		OP_NP,
-		OP_PASSNEXT,
+		0,
 		sizeof (int),
 		0
 	}
@@ -867,7 +867,7 @@ static void tl_fill_option(uchar_t *, cred_t *, pid_t, int, cred_t *);
 static int tl_default_opt(queue_t *, int, int, uchar_t *);
 static int tl_get_opt(queue_t *, int, int, uchar_t *);
 static int tl_set_opt(queue_t *, uint_t, int, int, uint_t, uchar_t *, uint_t *,
-    uchar_t *, void *, cred_t *, mblk_t *);
+    uchar_t *, void *, cred_t *);
 static void tl_memrecover(queue_t *, mblk_t *, size_t);
 static void tl_freetip(tl_endpt_t *, tl_icon_t *);
 static void tl_free(tl_endpt_t *);
@@ -904,7 +904,6 @@ optdb_obj_t tl_opt_obj = {
 	tl_default_opt,		/* TL default value function pointer */
 	tl_get_opt,		/* TL get function pointer */
 	tl_set_opt,		/* TL set function pointer */
-	B_TRUE,			/* TL is tpi provider */
 	TL_OPT_ARR_CNT,		/* TL option database count of entries */
 	tl_opt_arr,		/* TL option database */
 	TL_VALID_LEVELS_CNT,	/* TL valid level count of entries */
@@ -2789,12 +2788,10 @@ tl_optmgmt(queue_t *wq, mblk_t *mp)
 	 * call common option management routine from drv/ip
 	 */
 	if (prim->type == T_SVR4_OPTMGMT_REQ) {
-		(void) svr4_optcom_req(wq, mp, cr, &tl_opt_obj,
-		    B_FALSE);
+		svr4_optcom_req(wq, mp, cr, &tl_opt_obj);
 	} else {
 		ASSERT(prim->type == T_OPTMGMT_REQ);
-		(void) tpi_optcom_req(wq, mp, cr, &tl_opt_obj,
-		    B_FALSE);
+		tpi_optcom_req(wq, mp, cr, &tl_opt_obj);
 	}
 }
 
@@ -6066,8 +6063,7 @@ tl_set_opt(
 	uint_t		*outlenp,
 	uchar_t		*outvalp,
 	void		*thisdg_attrs,
-	cred_t		*cr,
-	mblk_t		*mblk)
+	cred_t		*cr)
 {
 	int error;
 	tl_endpt_t *tep;
diff --git a/usr/src/uts/common/io/warlock/ibcm.wlcmd b/usr/src/uts/common/io/warlock/ibcm.wlcmd
index b4ae04a925..e66149c4fd 100644
--- a/usr/src/uts/common/io/warlock/ibcm.wlcmd
+++ b/usr/src/uts/common/io/warlock/ibcm.wlcmd
@@ -66,11 +66,7 @@ root	ibt_get_src_ip
 root	ibt_ofuvcm_get_req_data
 root	ibt_ofuvcm_proceed
 
-root	ibcm_arp_timeout
 root	ibcm_arp_get_srcip_plist
-root	ibcm_arp_lrput
-root	ibcm_arp_lwsrv
-root	ibcm_arp_lrsrv
 root	ibcm_arp_get_ibd_insts_cb
 
 # callback entry points from ibmf