Our understanding of how neural networks learn from structured data is caught between two extremes: theory largely relies on high-dimensional random data, which lacks any structure, while experiments rely on real-world benchmarks, whose structure is not understood. This talk takes the middle road and works with a synthetic data model that is simple enough to be controlled yet rich enough to be informative. Sequences are generated by a branching process on a tree, so that elements inherit nested correlations across all scales, in a manner reminiscent of language or images, while the Bayes-optimal predictor remains known exactly. This provides a ground truth against which trained models can be compared, both in what they learn and in what they produce. In a first part, we ask how the structure is learned. Using a filtering procedure that controls the range of correlations, we show that transformers recover the exact inference algorithm, incorporating larger scales sequentially, in time during training and in space across attention layers. In a second part, we ask whether the structure, once learned, can be reproduced. Turning to diffusion models, we identify a phase of biased generalization, during which the test loss keeps decreasing while generated samples drift anomalously close to the training data. We quantify this bias by comparing networks trained on disjoint datasets, demonstrate its presence on real images, and use the exact scores available on tree-based data to characterize its onset. The mechanism is again sequential: coarse structure is learned early and independently of the specific data, while finer features are resolved later in a way that increasingly depends on individual training samples. As a consequence, early stopping at the test loss minimum may be insufficient for privacy-critical applications.