Skip to content
Tech Interview Prep
Technical interview guide

Neural Network Fundamentals

Forward pass, backpropagation, and the activation functions that make deep networks work.

Read
27 min
Practice MCQs
25
Interview Q&A
25
Edition
v3
Editorial status
Reviewed

Scope: Google ML Crash Course neural-network guidance, TensorFlow Keras guides, and PyTorch stable reproducibility guidance accessed 2026-08-31..

Curated: · Written: · Reviewed:

25 questions

MCQ-1A dense layer maps a 3-element input to 2 outputs. Its weight matrix W has one row per output unit and its bias vector b has one entry per output unit: W = [[1, 0, −1], [0.5, 2, 1]], b = [0.5, −0.5]. For the input x = [2, −1, 0.5], what is the pre-activation (the value fed into the activation function) of the second output unit?
MCQ-2A team trains a 4-layer MLP (50 → 128 → 128 → 1) on a regression task with no activation functions between the dense layers. It converges to a validation MSE of 0.83, identical to a linear regression baseline. After inserting a ReLU after each hidden layer and retraining, validation MSE drops to 0.41. What explains the original plateau?
MCQ-3A team profiles a 30-layer MLP and finds the gradient at layer 1 is about 1e-4 of the gradient at the output with sigmoid hidden units. They want to swap in an activation whose per-element forward cost is a single comparison and select, whose derivative is 1 for positive pre-activations and 0 for negative ones, and whose negative outputs are exactly 0 so a downstream sparse kernel can skip them. Which activation should they install?
MCQ-7A one-hidden-unit network trains on a single example. Input x = 3. The hidden pre-activation is z = w1·x + b1 with w1 = 2 and b1 = −1, passed through ReLU to give h. The output is ŷ = w2·h + b2 with w2 = 0.5 and b2 = 0. The loss is L = (ŷ − t)² with target t = 1. One backward pass runs. What value does backprop assign to ∂L/∂w1?
MCQ-11An MLP with ten weight layers is profiled at initialization. Each layer scales the backpropagated gradient by a fixed factor: the six layers nearest the input multiply it by 0.5, and the four layers nearest the output multiply it by 2.0. The gradient of the loss with respect to the network's output has norm 1.0. What is the norm of the gradient of the loss with respect to the network's input?
MCQ-16A feed-forward network takes a 64-dimensional input whose elements each have variance 1 and passes it through dense layers of width 64, 256, and 1024 in that order, with no activation functions between them. Every weight is drawn i.i.d. from a zero-mean Gaussian with standard deviation 0.1, independent of the layer's width, and all biases are zero. What is the variance of the pre-activations at the output of the width-256 layer?
MCQ-8A training loop runs SGD with lr = 0.1, momentum = 0, and weight_decay = 0. Each iteration calls loss.backward() and optimizer.step(), but the loop never calls optimizer.zero_grad(). For one particular weight, backward() computes a gradient of 0.5 on each of the first three iterations. What update does the optimizer apply to that weight on the third iteration?
MCQ-9A model is trained with plain SGD (no momentum) on a locally quadratic loss L(w) = 50w², so dL/dw = 100w. The learning rate is 0.05, the weight starts at w = 1, and the run is in float32. Which outcome matches the next 30 steps?
MCQ-10A hidden layer has two tanh units. Every weight and bias starts at 0. The input is x = 1, the output weights are v1 = v2 = 1, the target is y = 1, the loss is 0.5·(out − y)², and SGD uses a learning rate of 0.1. After one update step, what are the two hidden weights, and what happens to the two units over the rest of training?
MCQ-4A team's 3-class ticket classifier ends in three independent sigmoid units. For a validation ticket, the logits are [2.0, 1.0, 0.0], and the sigmoid outputs are [0.880, 0.731, 0.500]. The team reports the predicted class's probability as 0.880. Under the output setup that matches single-label multiclass classification, what is the probability of the predicted class for this ticket?
MCQ-5A moderation model assigns policy labels from a fixed set of 12, and 38% of labelled items carry two or more. It is trained as a single softmax over the 12 labels with categorical cross-entropy. On validation, a label that usually co-occurs with another has recall 0.21 on co-occurring items but 0.86 on items where it appears alone, and its softmax score on co-occurring items sits between 0.03 and 0.08. Which change addresses this?
MCQ-6A float16 classifier outputs logits [90.0, 0.0, 0.0] for one example; the target class is index 1. The training loop computes exp(logits) / sum(exp(logits)) then -log(p[1]) and gets inf. Replacing those two lines with a fused cross-entropy that takes logits yields what?
MCQ-12A ResNet is trained with SGD (lr 0.1, momentum 0.9). Over the first 5,000 steps the global gradient norm sits near 2.5, with occasional spikes to about 300 that drive the loss from 1.8 to 7.0. The team adds global-norm clipping with max_norm = 1.0. The spikes stop, but after 10,000 further steps the loss is 0.6 above the unclipped baseline. Which explanation fits?
MCQ-13A team trains a ResNet on 50,000 images with SGD (momentum 0.9), batch size 100, LR 0.1, for 30 epochs, reaching 78% top-1. To use the GPUs better they raise the batch to 500, keep the LR at 0.1 and the schedule at 30 epochs, and get 74%. They cannot afford more than 30 epochs. Which change is most likely to recover the 78%?
MCQ-14A BatchNorm layer in a ResNet has running_mean = 0.5 and running_var = 2.0 after training (eps = 1e-5). A serving process batches two requests together and calls the model without switching to eval mode. The two pre-normalization activations for a given channel are 3.0 and 5.0. What does the layer output for the activation 3.0, before the affine scale and shift?
MCQ-15A team hand-rolls dropout for a 100-unit hidden layer: during training each unit is zeroed with probability 0.5 and the survivors are left unscaled, with each surviving activation averaging 1.0. At inference all 100 units are active and nothing is scaled. The next layer's weights were fit against the training-time signal. What does that next layer see at inference?
MCQ-24A team trains an image classifier for 200 epochs, logging mean cross-entropy after each epoch on the training split and on a held-out validation split. Preprocessing is identical for both splits and the model is in eval mode during validation. Which pattern in the two logged curves most strongly indicates overfitting?
MCQ-23A team gradient-checks a smooth custom layer in float64 using central differences. The maximum relative error is 2e-5 at eps=1e-2, 4e-11 at eps=1e-5, and 2e-7 at eps=1e-9. What does this pattern indicate?
MCQ-22A Keras model begins with `Dense(8, activation='relu', input_shape=(6,))` and ends with `Dense(3)`. It trains normally on batches of shape (32, 6). A teammate swaps in a new dataset whose batches have shape (32, 9), and training stops immediately with `ValueError: Input 0 of layer dense is incompatible with the layer: expected axis -1 of input shape to have value 6 but received input with shape (32, 9)`. What is the correct diagnosis?
MCQ-17A ranking model must apply the same encoder weights to a query embedding and a document embedding, then emit two outputs: a relevance score trained with MSE and a click probability trained with binary crossentropy. The serving team reloads models from a saved config and cannot ship custom Python classes. Which API should the model be built with?
MCQ-18A job checkpoints every 500 steps as `{'model': model.state_dict(), 'opt': opt.state_dict(), 'loader': loader.state_dict()}`; the global step is not saved. The LR schedule is `lr = 3e-4 * 0.5 ** (step // 20000)`, evaluated from the current step, and checkpoints are taken at epoch boundaries. The job is preempted at step 40,000, where the run's LR was 7.5e-5 and training loss 1.38. It resumes with `step` reset to 0. Within 500 steps the loss climbs to 1.9 and stays near 1.85 for the next 5,000 steps. What is the cause?
MCQ-19A team trains a 400M-parameter float32 model with Adam. The training checkpoint on disk is 4.8 GB. They export a serving artifact from that same checkpoint, it is also 4.8 GB, and the inference server's 2 GB memory limit makes it fail to load. Which export is correct, and what should it weigh?
MCQ-20A team pins the Python, NumPy and PyTorch seeds, sets cudnn.benchmark=False and cudnn.deterministic=True, uses num_workers=0 and a single CUDA stream, and gets bit-identical loss curves on repeated runs on workstation A. They copy the identical code, data, library and CUDA versions to workstation B, whose GPU is a different model, and the curves diverge around step 200. Which explanation fits?
MCQ-21A team trains a 1.3B-parameter transformer on 8x A100 80GB GPUs with PyTorch 2.3 and CUDA 12.1. Two runs with the same seed and the same data end with losses that differ by 0.004, so they call torch.use_deterministic_algorithms(True) and set CUBLAS_WORKSPACE_CONFIG=:4096:8. Runs now match bitwise, but step time rises from 210 ms to 265 ms. Profiling shows the extra time sits in the kernel that accumulates per-token gradients into the embedding table, and the gap grows with the number of tokens per step while the GEMM kernels are unchanged. Which explanation fits?
MCQ-25A team exports a two-tower retrieval model to ONNX and runs a parity harness: 2,000 golden queries against the same 10,000-document corpus, both paths producing a full score vector per query, with the harness comparing the two paths' outputs to each other and treating score differences below 1e-5 as matches. The largest score difference anywhere in the run is 3e-7. The top-1 document id differs on 60 queries, and in each of those 60 the two top-scoring documents have identical scores in both paths. What is the most likely cause of the 60 disagreements?