Fix frozen pipes as soon as you find them as ice presents a bigger potential problem than just a temporary loss of water service; if the ice stays in there and continues freezing long enough, it will expand and burst the pipe. It’s important that you quickly find and thaw out the ice clog before that happens.
Press your hand along the exterior of the pipe to confirm that it's very cold. There may or may not be frost formations on the outside of the pipe. (Note: If no faucet anywhere in the house works, then the ice clog is likely in the main feed that comes into the basement, before it gets to the main shutoff valve or any faucet.)
Watch for any trickle of water from the faucet. Once it starts, the flow should gradually increase. Continue until it seems to be flowing freely and starts to diminish again (since all the water in the line will eventually drain).