So I understand that the subnet mask provides information about the length of the routing prefix (NID). It can be applied to a given IP address to extract the most significant bits allocated for the routing prefix and "zero out" the host identifier.
But why do we need the bitwise AND for that, specifically? I understand the idea, but would it not be easier to only parse the IP address ~~string~~ sequence of bits only for the first n bits and then disregard the remainder (the host identifier)? Because the information necessary for that is already available from the subnet mask WITHOUT the bitwise AND, e.g., with 255.255.255.0 or 1111 1111.1111 1111.1111 1111.0000 0000, you count the amount of 1s, which in this case is 24 and corresponds to that appendix in the CIDR notation. At this point, you already know that you only need to consider those first 24 bits from the IP address, making the subsequent bitwise AND redundant.
In the case of 192.168.2.150/24, for example, with subnet mask 255.255.255.0, you would get 192.168.2.0 (1100 0000.1010 1000.0000 0010.0000 0000) as the routing prefix or network identifier when represented as the first address of the network, however, the last eight bits are redundant, making the NID effectively only 192.168.2.
Now let's imagine an example where we create two subnets for the 192.168.2.0 network by taking one bit from the host identifier and appending it to the routing prefix. The corresponding subnet mask for these two subnets is 255.255.255.128, as we now have 25 bits making up the NID and 7 bits constituting the HID. So host A from subnet 192.168.2.5/25 (HID 5, final octet 0000 0101) now wants to send a request to 192.168.2.133/25 (HID 5, final octet 1000 0101). In order to identify the network to route to, the router needs the NID for the destination, and it gets that by either discarding the 7 least significant bits or by zeroing them out with a bitwise AND operation. Now, my point is, for identifying the network of which the destination host is part of (in this case, the host is B), the bitwise AND is redundant, is it not?
So why doesn't the router just store the NID with only the bits that are strictly required? Is it because the routing table entries are always of a fixed size of 32 bits for IPv4? Or is it because the bitwise AND operation is more efficiently computable?
Probably because it’s only four bytes of data, and counting/extracting bits takes more cpu time than one AND operation.
Most CPU’s are optimised to work with whole integers (32/64 bit) rather than individual bits.
If memory was a serious concern you could compress it down to one byte as a ‘number of 1s’ counter at the cost of additional cpu operations, but because 3 extra bytes is such a small amount of data, this memory/time trade off isn’t worth it in most systems.
It’d be useful if you wanted to compress some data logs or something with many subnet masks though.