What are the effects of ddr4 RAM timings? The essence and purpose of RAM timings. Dependence of performance on memory frequency

IntroductionWe don’t often turn to testing the dependence of the performance of modern high-level platforms on the characteristics of the memory subsystem. This is not such a burning topic that is of interest to the broad masses of users. Everyone has long been accustomed to the fact that the operating frequency of DDR3 SDRAM and its timings do not have a noticeable effect on performance, and therefore not too much attention is paid to the choice of memory. The selection of memory modules when assembling new systems in most cases occurs on a residual basis, and even many enthusiasts are guilty of this approach. In fact, the only memory characteristic that is seriously thought about is its capacity. Everyone knows that a lack of RAM can cause applications and the operating system to swap, and this ultimately causes deterioration in the responsiveness of the computer. But it’s not common to think that the speed of operation of memory modules can be significantly affected by the speed specifications of memory modules.

This situation did not arise out of nowhere. Previously, not too much depended on DDR3 SDRAM parameters such as its frequency and latency. This was explained by several reasons. Firstly, some time ago processors acquired significant amounts of cache memory, equipped with effective data prefetching algorithms that effectively hide the real speed of information exchange with memory from programs. Secondly, the speeds and latencies of the DDR3 SDRAM options available on the market until recently actually did not differ too much. And thirdly, applications that move really large amounts of information were rarely seen by ordinary users. As a result of all this, the judgment arose that fast DDR3 SDRAM is a kind of status product for perfectionists, and ordinary people do not need it.

However, this opinion, which a couple of years ago could be considered quite reasonable, is today somewhat outdated, and it is not difficult to criticize it. The main thing is that today's applications have changed greatly in their structure; now they operate with much larger volumes of information than before. Processing digital photographs measuring several tens of megapixels has become popular, many users have engaged in creative work with video files shot in FullHD or even 4K resolution, and modern 3D games have reached the point of interacting with truly colossal volumes of texture information. Such data arrays can no longer fit into the processor cache, the capacity of which, by the way, has practically stopped growing over the past few years.

The memory available on the market, on the contrary, has significantly expanded its variety of types. The frequencies of DDR3 SDRAM available on the shelves of computer stores today differ by more than two times, so just by selecting certain modules, you can vary the bandwidth of the dual-channel memory subsystem within a very wide range: from 21 to 47 GB/s and even more. We should not forget that the latest Haswell processors have become noticeably more productive than their predecessors, and, consequently, their need for quickly obtaining data for processing has increased. Therefore, it is quite possible to expect that the critical threshold, up to which the speed of slow memory like DDR3-1333 or DDR3-1600 was quite enough for the vast majority of needs, has finally been passed. In other words, there are plenty of arguments in favor of studying the dependence of the real performance of modern systems on the parameters of the memory subsystem.

But there is another reason why today we decided to turn to DDR3 SDRAM tests with various frequencies and timings. The fact is that the opportunity to study the intricacies of how such memory works using actual material is now being given to us almost for the last time. Starting from the second half of this year, faster, more economical and advanced DDR4 SDRAM will gradually begin to be introduced into the desktop market. For the first time, its support will appear in Haswell-E processors, and then, in 2015-2016, DDR4 SDRAM will appear in the promising LGA 1151 platform and Skylake processors. In other words, DDR3 SDRAM tests are not only long overdue, but there is no way to delay them any longer. Therefore, we will talk now about what different DDR3 SDRAM can offer for platforms based on the most popular Haswell processors at the moment.

Haswell Memory Controller Features

At first glance, the memory controller of modern processors for the LGA 1150 platform, known under the code name Haswell, is not particularly different from the memory controllers of their predecessors - Sandy Bridge and Ivy Bridge. The evolution of memory algorithms in Intel processors was long and multi-stage. But in the latest generations of CPUs, ideological development seems to have come to an end - modern technologies for interacting with DDR3 memory are not only well optimized, but honed to perfection. The main step that set modern Intel controllers head and shoulders above other solutions was the introduction of the Ring Bus to connect all the structural units in the processor design, and this was done back in Sandy Bridge. Thanks to the ring bus, all computing and graphics resources of the processor received fast and equal access to both the third level cache and the memory controller. As a result, the practical bandwidth of the memory subsystem has increased significantly, and its latencies have decreased.

However, the foundation of the memory controller in Haswell, which was previously laid in the form of a ring bus, has nevertheless undergone some important changes. The fact is that in earlier processor designs, the ring bus, together with the third level cache, worked synchronously with the CPU cores. And this created some inconveniences when the processor switched to energy-saving states: the L3 cache and ring bus could reduce their performance along with the computing cores, despite the fact that these resources remained in demand by the graphics core. To prevent such unpleasant collisions from occurring again, in Haswell the Ring Bus and L3 cache were allocated into a separate domain and received their own independent frequency.



The introduction of the possibility of asynchronous clocking of the ring intraprocessor bus, naturally, introduced inevitable delays in operations with the L3 cache and memory controller, but Intel developers tried to counteract the slowdown of the memory subsystem with various microarchitectural improvements. Thus, the third-level cache received two parallel queues to process requests for different purposes, and the memory controller had larger queues and an improved scheduler.

In addition, the asynchrony of the ring bus, L3 cache and memory controller does not always manifest itself. In reality, if you do not take into account energy-saving states, their frequency almost always coincides with the frequency of the computing cores. Discrepancies arise only in two situations: when the processor switches to turbocharged modes, or during overclocking. But even in these cases, the frequency of the L3 cache and the intraprocessor bus remains close to the frequency of the computing cores, and the difference between them usually does not exceed 300-500 MHz, which, as practice shows, has almost no effect on the final performance.

When directly comparing the performance of the Haswell memory controller and the Ivy Bridge memory controller, it turns out that with the same settings, the newer version provides generally similar throughput and latency. For example, this can be seen in the test results in AIDA64.



Ivy Bridge, 4 cores, 4.0 GHz, DDR3-1600 9-9-9-24-1N



Haswell, 4 cores, 4.0 GHz, DDR3-1600 9-9-9-24-1N


However, as can be seen from the results presented, despite all the efforts of Intel engineers, the memory in Haswell still works a little slower than in LGA 1155 systems of the previous generation, based on the Ivy Bridge processor. And if the difference in practical throughput is almost imperceptible, then the latency of the Haswell memory subsystem is approximately 9 percent higher. This is the price for asynchrony.

The second significant change regarding the operation of the memory subsystem in LGA 1150 systems relates to the design of motherboards. Intel's DIMM slot reference design is now based on a T-topology, which equalizes the rights of the DIMM slots attached to each channel. This improves the stability of the memory controller and makes it compatible with a wider range of different memory modules and their configurations. What is especially nice here is that the memory controller of Haswell processors is able to support high-speed operating modes even when using four double-sided modules installed in all available DIMM slots. Considering that the maximum capacity of DDR3 memory sticks available on the market is 8 GB, the LGA 1150 platform can ensure smooth operation of 32 GB overclocking memory arrays with high frequencies and low latencies.

Otherwise, everything remains the same as before. Haswell's memory controller is dual-channel, capable of operating in both symmetrical dual-channel and single-channel modes. There is also support for Flex Memory technology, which allows the use of dual-channel access in asymmetric configurations, when the volumes and characteristics of the modules installed in different memory channels do not match.

As in Ivy Bridge processors, Haswell's DDR3 SDRAM frequency varies in increments of 266 or 200 MHz, which gives some flexibility in choosing modes and seriously expands the variety of DDR3 SDRAM operating frequencies available to the controller. At the same time, the controller formally supports only DDR3-1333 and DDR3-1600 SDRAM, however, all the improvements made in it allow the free use of memory operating at significantly higher frequencies in the LGA 1150 platform. Thus, the available set of multipliers for the memory frequency allows you to activate modes up to DDR3-2933, and such high-speed modes are actually achievable, no stability problems are observed when using them.

If we add to this the new ability to overclock the Haswell base frequency from 100 to 125 MHz, then the memory frequencies available for use will increase to 3666 MHz. Moreover, on the Internet you can find a lot of evidence that even in this state in LGA 1150 systems, selected overclocker memory can be quite functional.



As you know, important changes at Haswell occurred with the power system. This processor has a built-in power converter that independently generates all the voltages necessary for CPU operation. Now only two voltages depend on the motherboard: the input voltage for the processor - Vccin and the voltage supplied to the power modules - Vddq. However, internal processor voltages, including the ring bus signal voltage and the supply voltage of the L3 cache and memory controller, are generated by the processor power circuit independently. This innovation freed the memory voltage from any restrictions, and in Haswell processors it is allowed to safely increase it above the level of 1.65 V. In other words, in LGA 1150 you can overclock the memory by changing its supply voltage as you like, without worrying about possible degradation processor memory controller.



Thus, the combination of innovations has made the new DDR3 SDRAM controller of Haswell processors not only highly efficient, but also well suited for working with overclocker memory modules. This means that enthusiasts have enormous freedom in choosing memory for LGA 1150 systems, which may well affect the final performance.

G.Skill F3-2933C12D-8GTXDG

Before moving on to the test results, a few words need to be said about the memory modules that made this study possible. In order to get the most complete picture of the dependence of performance on the parameters of the memory subsystem, we needed a set of DDR3 SDRAM modules with the highest possible frequency. Such memory kits are characterized by the greatest flexibility. They do not have to be operated at the cosmic frequencies declared for them; it’s just that for their flagship DDR3 overclocking strips, manufacturers select the most advantageous chips that maintain stability over the widest range of settings. If we take into account the fact that the Haswell memory controller is capable of providing modes up to DDR3-2933, this is exactly the DDR3 we wanted to get for testing.

At the moment, only a few manufacturers have mastered serial production of DDR3-2933 SDRAM overclocking kits. These include: ADATA, Corsair, Geil and G.Skill. And it was the last company from this list that responded to our request to provide us with its flagship product for testing, thanks to which we received the G.Skill TridentX F3-2933C12D-8GTXDG kit, consisting of a pair of 4 GB high-speed “slats”. This memory is designed to operate at a frequency of 2933 MHz with nominal timings of 12-14-14-35-2N, however, as we were able to verify during the tests, in reality it is capable of operating in a slightly higher speed mode when the Command Rate is set to 1N.



The specifications of this overclocking memory kit are as follows:

The dual-channel kit consists of two modules of 4 GB each;
Nominal frequency: 2933 MHz;
Timings: 12-14-14-35-2N;
Operating voltage 1.65 V.

The modules included in the kit in question are covered on both sides with proprietary two-color red and black aluminum heat dissipators of the TridentX series. A special feature of these radiators is their two-tier articulated design. Unlike many other manufacturers, G.Skill heeded numerous user complaints that tall radiators do not fit well with massive CPU coolers. Therefore, the TridentX series radiators are made collapsible. Their upper (red) part can be easily removed after unscrewing the two mounting screws, and in the “lightweight” version the height of the modules is reduced from 54 mm to only 39 mm. In this case, there are no problems with mechanical compatibility with massive CPU coolers, and the remaining part of the radiator is quite enough to effectively remove heat from the memory chips.



To ensure ease of installation and configuration, G.Skill TridentX F3-2933C12D-8GTXDG modules support XMP 1.3 technology. The only prepared XMP profile contains the frequency and delays declared in the specification. If we add to this the flexibility and ease of configuration of the memory controller of Haswell processors, practically running this memory at a frequency of 2933 MHz is not difficult. The formula “stick it in and go” is perfectly applicable in this case. To ensure stable operation of the memory controller, most likely, even an additional increase in any intra-processor voltages will not be required. However, just in case, to ensure maximum compatibility, the SPD of the modules in question contains configurations for various DDR3-1333 variants.



The G.Skill high-speed memory is based on Hynix H5TQ4G83MFR chips, which are very popular among overclockers, which are mounted on a specially designed eight-layer printed circuit board. This design, characterized by excellent overclocking potential and low heat dissipation, has proven itself, and its use in memory aimed at conquering ultra-high frequencies is quite natural. Practical testing showed: in an LGA 1150 system, the G.Skill TridentX F3-2933C12D-8GTXDG kit can operate excellently at a frequency of 2933 MHz with timings of 12-14-14-35-1N.



It must be said that the G.Skill TridentX F3-2933C12D-8GTXDG modules are specifically aimed at systems with Haswell processors, which are based on Intel Z87-based motherboards. Memory frequency DDR3-2933 MHz is currently available only in such platforms. At the same time, the modules in question have a fairly extensive list of motherboards tested for compatibility. In fact, we can say that the use of such memory does not impose any restrictions on the choice of motherboard. Most mid- and high-price motherboard models from all leading manufacturers can work stably with the G.Skill TridentX F3-2933C12D-8GTXDG kit, which is its important advantage.



In fact, the only disadvantage of high-speed DDR3 SDRAM kits like the one under review is their significant price. For example, the G.Skill TridentX F3-2933C12D-8GTXDG kit costs several times more than a similar dual-channel DDR3-1866 kit. So the validity of choosing this option from the point of view of a rational buyer is a big question. This is an exclusive offer for performance enthusiasts.

Description of test systems

In the preparation of this material, we used the LGA 1150 platform, built on a modern motherboard with an Intel Z87 chipset, into which we installed an overclocker Core i5-4670K processor with a Haswell design. However, the main role in studying the dependence of performance on the settings of the memory subsystem went to the high-speed memory kit G.Skill F3-2933C12D-8GTXDG standard DDR3-2933, provided to us for this testing by the manufacturer.

In general, the following hardware and software components were involved in testing:

Processor: Intel Core i5-4670K, overclocked to 4.4 GHz (Haswell, 4 cores, 6 MB L3);
CPU cooler: NZXT Havik 140;
Motherboard: Gigabyte Z87X-UD3H (LGA1150, Intel Z87 Express).
Memory: 2x4 GB, DDR3-2933 SDRAM, 12-14-14-35 (G.Skill TridentX F3-2933C12D-8GTXDG).
Graphics card: NVIDIA GeForce GTX 780 Ti (3 GB/384-bit GDDR5, 876-928/7000 MHz).
Disk subsystem: Intel SSD 520 240 GB (SSDSC2CW240A3K5).
Power supply: Corsair AX760i (80 Plus Platinum, 760 W).

Testing was performed on the Microsoft Windows 8.1 Enterprise x64 operating system using the following set of drivers:

Intel Chipset Driver 9.4.0.1027;
Intel Management Engine Driver 9.0.2.1345;
Intel Rapid Storage Technology 12.9.0.1001;
NVIDIA GeForce Driver 334.89.

Note that in this testing we used a Haswell processor overclocked to 4.4 GHz. The fact is that an abnormal increase in clock frequency additionally increases performance and allows you to get a more pronounced picture of the dependence of performance on the parameters of the memory subsystem.

Frequency vs timings

Every time we talk about the optimal choice of memory, sooner or later the question arises about what we should strive for first: to increase the frequency of the memory subsystem or to reduce its latency. However, this time we will avoid detailed tests of DDR3 SDRAM modules, which differ only in timings. The fact is that with the release of each new platform, the impact of delays on overall performance has decreased, and by now it has perhaps already passed the critical point. Of course, the dependence of performance on timings can still be noticed, but compared to the effect that changing the DDR3 SDRAM frequency has on system performance, it has become insignificant.

There are two main reasons for this. Firstly, as the frequency of memory operation increases, its minimum latency in any case increases, and against this background, the relative magnitude of the increase in variable delays becomes less and less noticeable. It is one thing to increase the timing by a couple of cycles from three to four (as was the case with DDR2 SDRAM), and another thing - from nine to ten (in the case of high-speed DDR3 SDRAM). In the first case, latency increases by 50-70 percent, and in the second - only by 20-22 percent. Accordingly, from a practical point of view, the difference between different timing options in modern memory is not nearly as significant as before. In addition, the timing circuit lost its original meaning due to the general improvement in the way processors work with memory. Multi-level caching used in modern processors, as well as prefetch algorithms, seriously masks the real latency of RAM, shifting the emphasis to its bandwidth.

In fact, manufacturers of overclocking memory kits have long realized that there is no need to race for low timings in high-frequency DDR3 SDRAM. Offers with a latency of 7-8 cycles have long disappeared from sale, and now on store shelves it is quite difficult to find DDR3 SDRAM modules with a CAS Latency parameter of less than 9-10 cycles. The number of proposals with ultra-high frequencies and long delays is steadily growing.

However, we would not like to leave unfounded the statements about the insignificant impact of timings on the performance of the memory subsystem in modern platforms built on Haswell processors. Therefore, we also conducted practical testing, in which we compared the actual performance of identical systems equipped with DDR3-1600 and DDR3-1867 SDRAM with different latencies.












The above graphs provide a clear illustration of all of the above. Increasing the memory frequency by 266 MHz turns out to be noticeably more effective than reducing all delays by 3-4 cycles. And even from the point of view of real latency, which reacts most sensitively to changes in delays, DDR3-1867 with fairly weak timings 10-10-10-29 turns out to be better than the unavailable DDR3-1600 with aggressive delays 7-7-7-21 . If we judge the performance of the memory subsystem based on real bandwidth indicators, then DDR3-1600 cannot compare with a slightly higher frequency version under any circumstances.

In other words, memory latency in modern systems has indeed become a very insignificant factor. Therefore, when choosing DDR3 SDRAM for Haswell processors, first of all you need to pay attention to its operating frequency, and low CAS Latency and other similar values ​​have virtually no effect on real performance. The same should be done when setting up and overclocking the system - first you should strive to increase the operating frequency of DDR3 SDRAM, and only then, if you really want to, minimize delays.

Dependence of performance on memory frequency

Let's move on to the main part of the study, for which everything was started: let's try to determine how much the parameters of the memory subsystem in the LGA 1150 platform affect the performance in ordinary commonly used applications. As shown above, DDR3 SDRAM timings in modern computer systems have an extremely insignificant effect even on the results of synthetic tests. Therefore, in detailed practical testing, we decided to abandon the comparison of memory subsystems with the same frequency, but different latencies, focusing on the more valuable task from a practical point of view of comparing DDR3 with different frequencies. Moreover, the majority of commercially available overclocking memory kits differ from each other only in latencies very rarely. The frequencies of DDR3 SDRAM currently available on the market are extremely varied and, wanting to cover the full range of options available for use, we tested a Haswell-based system with various types of memory, starting with DDR3-1333 and ending with DDR3-2933 SDRAM. In this case, the delays were set according to the most popular scheme for each frequency. More specifically, this means that tests were carried out with the following variants of dual-channel DDR3 memory:

DDR3-1333, 9-9-9-24-1N;
DDR3-1600, 9-9-9-24-1N;
DDR3-1866, 9-10-9-28-1N;
DDR3-2133, 11-11-11-31-1N;
DDR3-2400, 11-13-13-31-1N;
DDR3-2666, 11-13-13-35-1N;
DDR3-2933, 12-14-14-35-1N.

Apart from the settings of the memory subsystem in the test platform, based on a quad-core Haswell generation processor overclocked to 4.4 GHz, absolutely nothing has changed.

Synthetic tests

We decided to start by measuring practical throughput and latency. For this purpose, the Cache and Memory benchmark from the AIDA64 4.20.2820 utility was used.









As can be seen from the results, by varying the operating frequency of DDR3 memory, you can achieve an almost twofold change in practical bandwidth. Which, in general, is quite natural: the frequency and theoretical bandwidth of DDR3-1333 and DDR3-2933 differ by more than two times. What is somewhat surprising is that the dependence of the results on frequency turns out to be far from linear. For some reason, the fastest memory modes do not provide maximum throughput. The best results are demonstrated by DDR3-2400 and DDR3-2666. A further increase in frequency entails a slight drop in the speed of data exchange with memory.

However, practical latency varies slightly according to a different law.



Latencies with increasing DDR3 SDRAM frequency decrease in any case, including the transition to the highest speed modes. Thus, overclocking DDR3-2666 and DDR3-2933 may be far from useless in terms of the performance of ordinary applications. To check this, let's turn to tests in real problems.

Comprehensive Performance

To analyze complex weighted average performance in common applications, we used the popular Futuremark PCMark 8 2.0 benchmark, and, more specifically, its three test traces: Home, which simulates typical Internet activity of home users, plus their work in text and graphics editors; Work, which simulates working with various office applications and the Internet; and Creative, which reproduces the behavior of advanced users who are interested in serious processing of photo and video content, 3D games, and also actively use the network to obtain information and communicate.









The results were clearly not in favor of fast DDR3 SDRAM variants. In synthetic memory tests, everything looked very nice, but Futuremark PCMark 8 2.0 paints a diametrically opposite picture. If you believe the performance indicators of this test, then those users who believe that over the past 10-15 years the speed parameters of the memory subsystem have not received sufficient significance are right. The differences in the performance of systems with fast and slow dual-channel DDR3 SDRAM do not exceed 1-2 percent.

However, we won't rely on the comprehensive test suite alone and will additionally look at performance in popular applications.

Tests in applications

In Autodesk 3ds max 2014, we measure the rendering speed of a specially prepared complex scene in mental ray.



The speed of the final rendering is greatly affected by memory frequency. More than doubling the bandwidth of DDR3 SDRAM allows you to get only a very frivolous advantage at the level of one percent.

Performance in the new Adobe Premiere Pro CC is tested by measuring the rendering time of a H.264 Blu-Ray project containing HDV 1080p25 video with various effects applied.



But here, when processing high-resolution video content, the situation is completely different. The difference in system performance with DDR3-1333 and DDR3-2933 reaches 8 percent, and it cannot be called noticeable. In other words, among modern tasks there are those for which memory speed plays a very significant role.

By the way, if you look at the results in more detail, it becomes obvious that the most beneficial memory type for Premiere Pro is DDR3-2400. A further increase in frequency no longer entails a noticeable increase in performance, but prices for DDR3-2666 and DDR3-2933 kits, on the contrary, are noticeably higher than those of slower products.

We measure performance in the new Adobe Photoshop CC using our own test, a creative reworking of the Retouch Artists Photoshop Speed ​​Test, which involves typical processing of four 24-megapixel images taken with a digital camera.



Among the applications that are sensitive to the parameters of the memory subsystem is Photoshop. The platform, equipped with high-speed dual-channel DDR3-2933 SDRAM, exceeds the operating speed of a similar platform with DDR3-1333 by 12 percent. The advantage of the “optimal choice”, DDR3-2400 over the ubiquitous DDR3-1600, is also clearly visible: it reaches 8 percent.

To measure the speed of processors when compressing information, we use the WinRAR 5.0 archiver, with which we archive a folder with various files with a total volume of 1.7 GB with the maximum compression ratio.



Archiving files is a task in which one could observe good scalability of performance depending on memory frequency even earlier, in the era of the popularity of processors for LGA 1155, LGA 1156 and even LGA 775 sockets. Nothing has changed now. Each 266-MHz step in the DDR3 SDRAM frequency increases the speed of the WinRAR archiver by 3-4 percent. Overall, DDR3-2933 allows the Haswell processor to achieve 23 percent higher performance than when the system is equipped with DDR3-1333.

To evaluate the speed of video transcoding into the H.264 format, the x264 FHD Benchmark 1.0.1 (64bit) test was used, based on measuring the time the x264 encoder encodes the source video into the MPEG-4/AVC format with a resolution of 1920x1080@50fps and default settings. It should be noted that the results of this benchmark are of great practical importance, since the x264 encoder underlies numerous popular transcoding utilities, for example, HandBrake, MeGUI, VirtualDub, etc. We periodically update the encoder used for performance measurements, and this testing involved version r2389, which supports all modern instruction sets, including AVX2.



But when transcoding high-resolution video, performance scalability depending on the parameters of the memory subsystem is not so noticeable. The advantage of DDR3-2400 over the commonly used DDR3-1600 is only 3 percent, while one 266-MHz step in memory frequency allows for about 1 percent faster transcoding. Moreover, after increasing the memory frequency beyond the 2400 MHz mark, the performance increase becomes even more elusive.

Gaming Performance

The most interesting part of our testing is measuring gaming performance. The fact is that modern 3D games are among the tasks that require fast memory, and we expect that when used in games, fast memory will be able to fully realize its advantages.

At the same time, the performance of current high-performance platforms in the vast majority of modern games is determined by the power of the graphics subsystem. That is why during testing we selected the most processor-dependent games, and measured the number of frames twice. The first pass of the tests was performed without turning on anti-aliasing and setting far from the highest resolutions. Such settings make it possible to evaluate how much fast memory gaming systems need in general. That is, they allow us to speculate about how platforms with different DDR3 SDRAM will behave in the future, when faster graphics accelerator options appear on the market. The second pass of performance measurement was performed with realistic settings - when selecting FullHD resolution and the maximum level of full-screen anti-aliasing. In our opinion, such results are no less interesting, since they answer the frequently asked question about what level of gaming performance can be obtained right now - in modern conditions.












When measuring frame rates in 3D games at a lower resolution, it turns out that modern shooters can easily be classified as tasks that are extremely responsive to the performance of the memory subsystem. As you can see from the results, memory frequency alone can increase performance by a third - this is exactly the situation observed in the new Thief. In other games, the impact of memory is less pronounced, but, nevertheless, the average difference in performance between a Haswell-based platform with slow DDR3-1333 and overclocking DDR3-2933 is about 20 percent. In other words, every 266 MHz increase in DDR3 SDRAM frequency increases gaming performance by 2-3 percent.

However, such impressive scalability was achieved largely due to the fact that we deliberately unloaded the graphics subsystem. If you set the maximum quality settings in games, the picture will look like this.












Here the impact of memory speed on performance is much less pronounced. If previously the difference in the performance of systems with fast and slow memory reached tens of percent, then choosing a high image quality reduces the maximum increase by approximately an order of magnitude. However, using the example of Thief, we can conclude that this situation is not typical for all games. There are situations in which the frequency of DDR3 memory can significantly affect performance even in modes with maximum quality settings. Thus, hardcore gamers looking to get the most out of their systems should not neglect fast memory. Situations where this particular component of the platform can have a noticeable impact on performance are by no means incredible.

conclusions

The performance of modern systems built on Haswell generation processors has demonstrated a fairly noticeable dependence on the parameters of the memory subsystem, and primarily on the frequency of the modules used. We can say with all certainty that the era when memory parameters had practically no effect on anything has already passed. Today, just by selecting the characteristics of the DDR3 SDRAM strips installed in the system, you can increase the operating speed by 20-30 percent.

True, the speed of the memory subsystem does not always have such a clear impact on the performance of applications. Among the common tasks solved by personal computers, there are both memory insensitive to performance and those for which fast DDR3 SDRAM is more than important. Summarizing the test results, we can say that you should think about choosing high-speed sets of DDR3 SDRAM modules in two cases: either when completing gaming systems, or when assembling home workstations aimed at processing high-resolution images and video.

At the same time, the main attention in choosing memory for upper-level LGA 1150 platforms should be paid to frequency (naturally, after making an informed decision about the required volume), and not to delays. The DDR3 SDRAM kits presented on store shelves differ little in latencies, but their frequencies differ by more than twice. And this is not without reason. As practice shows, it is the DDR3 SDRAM frequency that has the primary impact on performance.

Modern systems built on Haswell processors are well prepared to work with high-speed DDR3. Memory clocking at frequencies up to 2933 MHz does not cause any problems and does not require any tweaks in configuration. Therefore, such a memory could well be recommended to all enthusiasts, if not for one thing. High-frequency memory is incredibly expensive, so it may only interest those rare buyers who do not have any budget restrictions. From the point of view of common sense, the most interesting option for high-performance systems has every chance of becoming DDR3-2400 SDRAM. The overclocking markup for such memory is not too high, and it provides a very decent performance increase compared to standard options like DDR3-1600. Moreover, a further increase in memory frequency, as tests show, gives a noticeably less effect, but the price after crossing the 2400 MHz mark soars astronomically.

Timings. Which is better RAM timing 9-9-9-24 or 9-9-9-27? p\s can you tell us in detail about Timings

  1. RAM timing is its response time, the lower the better.
  2. Everything is simple here - the higher the number, the worse, but here everything depends on the tire
    if 9-9-9-24 is a 1600 bus and 9-9-9-27 is 1866 then they are the same :)
  3. What timing is better for 800 MHz RAM? 6 or 6-6-6-18
  4. Another name for this term is latency, CAS Latency (CAS Latency = CL), that is, the time delay of the signal. Usually these time delays are called timings and for brevity they are written in the form: 2-2-2 (for example). These are the following parameters recorded in order: CAS Latency, RAS to CAS Delay and RAS Precharge Time. They can take values ​​from 2 (Kingston HyperX, OCZ line of memory modules) to 9. The throughput of the processor-memory section and, as a consequence, the performance of the main components of the system largely depend on them.

    Case Study: A system with 100 MHz memory with 2-2-2 timings has approximately the same performance as the same system with 112 MHz memory but with 3-3-3 timings. In other words, depending on latency, the performance difference can be as much as 10%.

    Measure of timings tact. Thus, each number in the 2-2-2 formula represents the signal processing latency, measured in system bus clocks. If only one number is specified (for example, CL2), then only the first parameter is implied, that is, CAS Latency. The rest are not necessarily equal to him! Practice shows that usually other parameters are higher, which means the memory is less productive (that is, it is a marketing ploy to indicate one timing in the specification, which does not represent the memory delays when performing other operations).

    Sometimes the timing formula for memory can consist of four numbers, for example 2-2-2-6. The last parameter is called DRAM Cycle Time Tras/Trc and characterizes the performance of the entire memory chip. It defines the ratio of the interval during which a row is open for data transfer (tRAS RAS# Active time) to the period during which a complete row opening and updating cycle is completed (tRC Row Cycle time), also called Bank Cycle Time. .

    Manufacturers usually supply their chips, on which the memory stick is based, with information about recommended timing values ​​for the most common system bus frequencies. You can view this information, for example, with the CPU-Z program.

    From the user's point of view, information about timings allows you to roughly estimate the performance of RAM before purchasing. DDR generation memory timings were given great importance because the processor cache was relatively small and programs accessed the memory frequently. Memory timings of the DDR3 generation are given much less attention, since modern processors (for example Intel Core DUO and Intel I5, I7) have relatively large L2 caches and are equipped (again relatively) with a huge L3 cache, which allows these processors to access memory much less frequently, and in some cases, the entire program is placed in the processor cache. Parameter name Designation Definition
    CAS LatencyCLThe delay between sending the column address to memory and the start of data transfer. The time required to read the first bit from memory when the desired row is already open.
    Row Address to Column Address DelayTRCDThe number of clock cycles between opening a row and accessing the columns in it. Time required to read the first bit from memory without an active TRCD + CL line.
    Row Precharge TimeTRPThe number of clock cycles between the command to precharge the bank (closing a row) and the opening of the next row. The time required to read the first bit from memory when another TRP + TRCD + CL line is active.
    Row Active TimeTRAS Number of cycles between the bank open command and the precharge command. Time to update the row. Superimposed on TRCD. Usually approximately equal to the sum of the three previous numbers.

Introduction

This article is a continuation of the popular material “The Impact of Memory on Computer Performance,” published on our website in April of this year. In that material, we experimentally established that the amount of memory does not greatly affect the performance of the computer, and in principle, 512 MB is quite enough for ordinary applications. After publication, our editors received many letters in which readers asked to suggest what kind of memory should be taken and whether it makes sense to buy more expensive memory, but with less capacity, and also asked to compare different types of memory.

And indeed, if in games the difference between the speeds of the same computer with 512 and 1024 MB of memory on board is negligible, maybe it’s worth installing 512 MB of expensive memory rather than 1024 MB of cheap? In fact, the performance of the same memory module is affected by delays, the so-called timings. Usually the manufacturer indicates them with a hyphen: 4-2-2-8, 8-10-10-12 and so on. Overclocked memory for enthusiasts usually has low timings, but is quite expensive. Ordinary memory, which simply works stably and does not promise speed records, has higher timings. This time we will find out what kind of timings these are, delays between what and what and how they affect computer performance!

Memory delays

With the industry's transition to the DDR-II standard, many users reported that DDR-II memory was not performing as fast as they would like. Sometimes even slower than the previous generation memory, DDR-I. This was due precisely to the large delays of the first DDR-II modules. What are these delays? They are usually labeled 4-4-4-12, four hyphenated numbers. They mean the following:

CAS Latency - RAS to CAS Delay - Row Precharge - Activate to Precharge

Let's try to clarify these notations. The memory bank consists of two-dimensional arrays. A two-dimensional array is the simplest matrix, each cell of which has its own address, row number and column number. To read the contents of a cell, the memory controller must first specify the row number and column number from which the data is read. To perform these operations, the controller must provide special signals to the memory.

RAS(Row Address Strobe) - a signal that determines the address of a row.

CAS(Column Address Strobe) - a signal that determines the column address.

CAS Latency(CAS) is the number of clock cycles from the moment the data is requested until it is read from the memory module. One of the most important characteristics of a memory module.

RAS to CAS Delay(TRCD) - delay between RAS and CAS signals. As we have already said, rows and columns are accessed separately from each other. This parameter determines the lag of one signal from another.

Row Precharge Delay(TRP) - the delay required to recharge the capacities of memory cells. Either the entire line is closed.

Activate to Precharge(TRAS) - strobe activity time. The minimum number of cycles between the activation command (RAS) and the recharge command (Precharge) or closing the same bank.

The lower these timings, the better: the memory will work faster with low latencies. But how much better and how much faster needs to be checked.

Memory for speed

The BIOS of modern motherboards allows you to manually change timing values. The main thing is that the memory modules support these values. By default, the timing values ​​are “hardwired” into the SPD chips of the modules and the motherboard automatically sets the values ​​recommended by the manufacturer. But nothing prevents enthusiasts from manually reducing latency by slightly overclocking the memory. This often leads to unstable operation. Therefore, in order to compare the impact of timings on speed, we will take a very fast memory and safely slow it down by changing certain delays.

This is a modern platform designed for use in high-performance computers. It is built on the Intel i925X chipset, which only supports DDR-2 memory, and uses PAT optimization technologies. This computer has very well designed ventilation, so we didn’t have to worry about overheating.

Test system

  • Intel Pentium 4 2.8 GHz (800 MHz FSB, 1024 Kb L2, LGA 775)
  • 80 Gb Maxtor DiamondMax 9 (7200 RPM, 8 Mb) S-ATA
  • SAPPHIRE RX600 PRO 128 Mb PCI Express
  • Windows XP Professional (Eng.) SP2
  • CATALYST 5.3

You need to test memory in different applications to see the difference in speed or, conversely, to show that there is none. Here we need the following tests:

    Synthetics

    • RightMark Memory Analyzer

      SiSoft Sandra 2005

    Emulation of real tasks

    • PCMark 2004 patch 120

  • RealWorld Test

Well, there are a lot of plans! Let's start with synthetics.


When overclocking a computer, we pay more attention to components such as the processor and video card, and sometimes we ignore memory, an equally important component. But fine-tuning the memory subsystem can further increase the speed of scene rendering in 3D editors, reduce the time required to compress a home video archive, or add a couple of frames per second in your favorite game. But even if you don't overclock, extra performance is never a bad thing, especially since with the right approach the risk is minimal.

Gone are the days when access to the memory subsystem settings in BIOS Setup was closed from prying eyes. Now there are so many of them that even a trained user can get confused with such variety, not to mention a simple “user”. We will try to explain as much as possible the actions necessary to improve system performance through the simplest settings of basic timings and, if necessary, some other parameters. In this material we will look at the Intel platform with DDR2 memory based on a chipset from the same company, and the main goal will be to show not how much performance will increase, but how exactly it needs to be increased. As for alternative solutions, our recommendations are almost completely applicable for DDR2 memory, and for regular DDR (lower frequencies and delays, and higher voltage) there are some reservations, but in general the principles of setting are the same.

As you know, the lower the delay, the lower the memory latency and, accordingly, the higher the operating speed. But you shouldn’t immediately and thoughtlessly reduce the memory settings in the BIOS, as this can lead to completely opposite results, and you will have to either return all the settings to their original settings or use Clear CMOS. Everything must be done gradually - changing each parameter, restarting the computer and testing the speed and stability of the system, and so on each time until stable and productive performance is achieved.

At the moment, the most current type of memory is DDR2-800, but it appeared recently and is only gaining momentum. The next type (or rather, the previous one), DDR2-667, is one of the most common, and DDR2-533 is already beginning to fade from the scene, although it is present on the market in sufficient quantities. There is no point in considering DDR2-400 memory, since it has practically disappeared from use. Memory modules of each type have a certain set of timings, and for greater compatibility with the existing variety of equipment, they are slightly overestimated. Thus, in the SPD of DDR2-533 modules, manufacturers usually indicate time delays of 4-4-4-12 (CL-RCD-RP-RAS), in DDR2-667 - 5-5-5-15 and in DDR2-800 - 5- 5-5-18, with a standard supply voltage of 1.8-1.85 V. But nothing prevents them from being reduced to increase system performance, and provided that the voltage is raised to only 2-2.1 V (which for memory will be within norms, but cooling still won’t hurt) it is quite possible to set even more aggressive delays.

We chose the following configuration as a test platform for our experiments:

  • Motherboard: ASUS P5B-E (Intel P965, BIOS 1202)
  • Processor: Intel Core 2 Extreme X6800 (2.93 GHz, 4 MB cache, FSB1066, LGA775)
  • Cooling system: Thermaltake Big Typhoon
  • Video card: ASUS EN7800GT Dual (2xGeForce 7800GT, but only “half” of the video card was used)
  • HDD: Samsung HD120IJ (120 GB, 7200 rpm, SATAII)
  • Drive: Samsung TS-H552 (DVD+/-RW)
  • Power supply: Zalman ZM600-HP

Two 1 GB DDR2-800 modules manufactured by Hynix (1GB 2Rx8 PC2-6400U-555-12) were used as RAM, which made it possible to expand the number of tests with different memory operating modes and timing combinations.

Here is a list of necessary software that allows you to check the stability of the system and record the results of memory settings. To check the stable operation of memory, you can use test programs such as Testmem, Testmem+, S&M, Prime95, is used as a utility for setting timings on the fly in the Windows environment. MemSet (for Intel and AMD platforms) and A64Info (for AMD only). Determining the justification of memory experiments can be done by an archiver WinRAR 3.70b(there is a built-in benchmark), program SuperPI, calculating the value of Pi, test package Everest(there is also a built-in benchmark), SiSoft Sandra etc.

The main settings are made in BIOS Setup. To do this, press the key during system startup Del, F2 or another, depending on the board manufacturer. Next, we look for the menu item responsible for memory settings: timings and operating mode. In our case, the required settings were in Advanced/Chipset Setting/North Bridge Configuration(timings) and Advanced/Configure System Frequency(operating mode or, more simply, memory frequency). In the BIOS of other boards, memory settings may be located in "Advanced Chipset Features" (Biostar), "Advanced/Memory Configuration" (Intel), "Soft Menu + Advanced Chipset Features" (abit), "Advanced Chipset Features/DRAM Configuration" (EPoX), "OverClocking Features/DRAM Configuration" (Sapphire), "MB Intelligent Tweaker" (Gigabyte, to activate the settings you must click in the main BIOS window Ctrl+F1) etc. The supply voltage is usually changed in the menu item responsible for overclocking and is designated as “Memory Voltage”, “DDR2 OverVoltage Control”, “DIMM Voltage”, “DRAM Voltage”, “VDIMM”, etc. Also, for different boards from the same manufacturer, the settings may differ both in name and location, and in quantity, so in each individual case you will have to refer to the instructions.

If there is no desire to raise the operating frequency of the modules (subject to the capabilities and support from the board) above its nominal value, then you can limit yourself to reducing delays. If yes, then you will most likely have to resort to increasing the supply voltage, as well as lowering the timings, depending on the memory itself. To change the settings, simply transfer the necessary items from the “Auto” mode to “Manual”. We are interested in the main timings, which are usually found together and are called as follows: CAS# Latency Time (CAS, CL, Tcl, tCL), RAS# to CAS# Delay (RCD, Trcd, tRCD), RAS# Precharge (Row Precharge Time, RP, Trp, tRP) and RAS# Activate to Precharge (RAS, Min.RAS# Active Time, Cycle Time, Tras, tRAS). There is also another parameter - Command Rate (Memory Timing, 1T/2T Memory Timing, CMD-ADDR Timing Mode) taking the value 1T or 2T (another value appeared in the AMD RD600 chipset - 3T) and is present on the AMD platform or in NVidia chipsets (in Intel logic it is locked at 2T). When this parameter is reduced to one, the performance of the memory subsystem increases, but its maximum possible frequency decreases. When trying to change the main timings on some motherboards, there may be some pitfalls - by disabling the automatic setting, we thereby reset the sub-timings (additional timings that affect both the frequency and performance of the memory, but not as significantly as the main ones), as, for example, on our test board. In this case, you will have to use the MemSet program (preferably the latest version) and look at the subtiming values ​​​​for each memory operating mode in order to set similar ones in the BIOS.

If the names of the delays do not coincide, then the “scientific poking method” works well here. By slightly changing the additional settings in BIOS Setup, we check with the program what, where and how has changed.

Now, for memory operating at a frequency of 533 MHz, you can try to set 3-3-3-9 or even 3-3-3-8 instead of the standard delays 4-4-4-12 (or some other option). If the system does not start with these settings, we raise the voltage on the memory modules to 1.9-2.1 V. Higher is not recommended, even at 2.1 V it is advisable to use additional memory cooling (the simplest option is to direct air flow from a conventional cooler to them ). But first you need to run tests with standard settings, for example in the very timing-sensitive WinRAR archiver (Tools/Benchmark and hardware test). After changing the parameters, we check again and, if the result is satisfactory, leave it as is. If not, as happened in our testing, then using the MemSet utility in the Windows environment (this operation can lead to either the system freezing, or, even worse, its complete inoperability) or using BIOS Setup, raise RAS# to CAS by one # Delay and test again. Afterwards, you can try to decrease the RAS# Precharge parameter by one, which will slightly increase performance.

We do the same for DDR2-667 memory: instead of the values ​​5-5-5-15 we set 3-3-3-9. When conducting tests, we also had to increase RAS# to CAS# Delay, otherwise the performance was no different from the standard settings.

For a system using DDR2-800, latencies can be reduced to 4-4-4-12 or even 4-4-3-10, depending on the specific modules. In any case, the selection of timings is purely individual, and it is quite difficult to give specific recommendations, but the examples given may well help you in fine-tuning the system. And don't forget about the supply voltage.

As a result, we tested with eight different options and combinations of memory operating modes and their latencies, and also included in the tests the results of an overclocker memory - Team Xtreem TXDD1024M1066HC4, operating at an effective frequency of 800 MHz with timings of 3-3-3-8. So, for the 533 MHz mode there were three combinations with timings 4-4-4-12, 3-4-3-8 and 3-4-2-8, for 667 MHz there were only two - 5-5-5-15 and 3 -4-3-9, and for the 800 MHz mode, as in the first case, three - 5-5-5-18, 4-4-4-12 and 4-4-3-10. The test packages used were: the memory subtest from the synthetic package PCMark05, the WinRAR 3.70b archiver, the Pi calculation program - SuperPI and the game Doom 3 (resolution 1024x768, High graphics quality). Memory latency was checked using the built-in benchmark of the Everest program. All tests were carried out in Windows XP Professional Edition SP2. The results presented in the diagrams are arranged according to operating modes.

As you can see from the results, the difference in some tests is insignificant, and sometimes even negligible. This is because the Core 2 Duo processor's 1066 MHz system bus has a theoretical bandwidth of 8.5 GB/s, which corresponds to the bandwidth of dual-channel DDR2-533 memory. When using faster memory, the FSB becomes the limiting factor in system performance. Reducing latency leads to increased performance, but not as noticeably as increasing memory frequency. If we used the AMD platform as a test bench, we could observe a completely different picture, which we will do next time if possible, but for now we will return to our tests.

In synthetics, the performance increase with reduced latency for each mode was 0.5% for 533 MHz, 2.3% for 667 MHz and 1% for 800 MHz. A significant increase in performance is noticeable when moving from DDR2-533 to DDR2-667 memory, but changing from 667 to DDR2-800 does not give such an increase in speed. Also, memory at a lower level and with low timings comes close to a higher-frequency version, but with nominal settings. And this is true for almost every test. For the WinRAR archiver, which is quite sensitive to timing changes, the performance indicator increased slightly: 3.3% for DDR2-533 and 8.4% for DDR2-667/800. Calculating the eight-millionth digit of pi treated the various combinations on a percentage basis better than PCMark05, although only slightly. The gaming application is not very fond of DDR2-677 with timings of 5-5-5-15, and only reducing the latter made it possible to bypass the slower memory (which, as it turned out, does not care what timings are) by two frames. Setting the DDR2-800 memory gave an increase of two more frames, and the overclocker version, which had a good gap in other tests, did not get much ahead of its less expensive analogue. However, in addition to the processor and memory, there is one more link - the video subsystem, which makes its own adjustments to the performance of the entire system as a whole. The memory latency result was surprising, although if you look closely at the graph, it becomes clear why the indicators are the way they are. Falling with increasing frequency and decreasing timings from the DDR2-533 4-4-4-12 mode, the latency has a “dip” on DDR2-667 3-4-3-9, and the latter mode is practically no different from the previous one except the frequency. And thanks to such low latencies, DDR2-667 easily outperforms DDR2-800, which has higher values, but the throughput of DDR2-800 allows it to still get ahead in real applications.

And in conclusion, I would like to say that despite the small percentage of performance increase (~0.5-8.5) that comes from reducing time delays, the effect is still present. And even when switching from DDR2-533 to DDR2-800, we get an average increase of 3-4%, and in WinRAR more than 20. So such “tuning” has its advantages and allows you to slightly increase system performance even without serious overclocking.

Greetings, dear readers! Today we will understand what timings in RAM mean and what this parameter affects. Indeed, suddenly, under this smart word, they are trying to sell us another dummy - for example, like megapixels in a mobile phone camera without sane optics?

From this article you will learn:

A little materiel

To understand timings - what they are and why they are needed, you should delve a little deeper into the mechanism of how the RAM works. A simplified diagram looks like this: RAM cells are arranged on the principle of two-dimensional matrices, which are accessed by specifying a column and a row.

Memory cells are essentially capacitors that can be charged or discharged, thus recording a one or a zero (I think everyone has long been aware that any computing device works with binary code).

By changing the voltage from high to low, a row access pulse (RAS) or column access pulse (CAS) is sent. Clock-synchronized signals are applied first to the row and then to the column. When recording information, an additional tolerance pulse (WE) is sent. Memory performance directly depends on the amount of data transferred per clock cycle.

There is one BUT: data is not transmitted instantly, but with some delay, which is also called latency. And, as we know, nothing is transmitted instantly - even photons of light have a finite speed. What about electrons trying to break through layers of silicon?

What do timings mean?

So, timing or latency is the amount of delay from receipt to execution of a command. There are several dozen types of them, as well as all kinds of sub-timings, but from a practical point of view they are of interest only to engineers and other major hardware specialists. For the average user, four types of timing are important, which are usually indicated when labeling the RAM:

  • tRCD – delay between RAS and CAS pulses;
  • tCL – delay from issuing a read or write command to the CAS pulse;
  • tRP – delay from processing a line to moving to the next one;
  • tRAS is the delay between the activation of the line and the start of processing.

Some manufacturers also indicate the Command rate - the delay between selecting a specific chip on the memory module and activating the line.

Marking

The measure of timing is the memory bus clock. In fact, these numbers allow you to generally evaluate the performance of the RAM bar even before purchasing it.

Typically, timings are indicated on the nameplate along with the memory type, frequency and other characteristics. For convenience, they are written as a set of numbers separated by a hyphen in the following order: tRCD-tCL-tRP-tRAS. For example, like this: 7–7–7–18.

However, not all manufacturers provide this information, so there is a possibility that if you disassemble the computer and remove the memory module, you will not find the required data. How can I find out the parameters of interest? In this case, programs that allow you to obtain complete information about the hardware will come to the rescue - for example, Speccy or CPU-Z.

And note that product descriptions in online stores often do not provide information about timings.

Therefore, if you decide to tinker with the hard drive and pick up an additional stick of RAM with absolutely identical timings in order to activate the dual-channel RAM mode (why do you need this), you will most likely have to go to a computer store and fool the seller (or find the information on the label yourself ).

Setting timings

Each RAM stick is equipped with an SPD chip, which stores information about recommended timing values ​​in relation to system bus frequencies. Typically, the computer automatically sets the optimal latency value, thanks to which the RAM will show better performance.

You can reassign timings in the BIOS. This is one of the favorite pastimes of overclockers and other computer wizards, who, with the help of all sorts of clever settings, can significantly increase the performance of any hardware. If you don't know what timings to set, it's better not to touch anything and select the automatic setting.

Naturally, when buying RAM, many are interested in the question of what will happen if different memory modules have different timings. In fact, nothing bad will happen - you simply will not be able to run the RAM in dual-channel mode.

There are known cases of complete incompatibility of memory modules, the joint use of which provokes the appearance of a “blue screen of death”, but here, in addition to latency, many additional parameters should be taken into account.

When you go shopping for a new memory stick, you may continue to doubt which timings are best. Naturally, those that are lower. However, the difference in latency figures is reflected in the difference in the figures on the price tag - with all other parameters being equal, a module with lower timings will cost more.

And if you have read my previous publications, then you probably still remember that I am indignant about the fossil DDR3 and encourage everyone to focus on the progressive DDR4 standard when assembling a computer.

It will also be useful for you to read articles on this topic and how the processor frequency and RAM frequency relate. For a deep dive, so to speak. To know everything at all.

With this, dear friends, I say to you “See you tomorrow.” Thank you for your attention and sharing this post on social networks.

Share with friends or save for yourself:

Loading...