Performance and Benchmarks

Test Systems and Methodology

Since this review focuses on a standalone Internet Gateway and its suitable Broadband Internet Service. The product is not reliant on the performance of client-connected computers so performance should be consistent regardless of the specification.

Client PCs used to benchmark performance and throughput comprised of mainly white-box and DELL systems with Intel Pentium 4 or AMD Athlon processors running Windows XP SP3.

For testing the performance and connectivity of the Wireless-G LAN, we used the following Wireless-G PCI desktop adapters in our test machines with the latest drivers available at time of testing

Netgear WG311T PCI Wireless Adapter– Atheros AR5001X chipset

D-Link DWL-G510 PCI Wireless Adapter – Ralink RT61 chipset

To test the performance of this Internet Gateway and the 3G Broadband Internet Connection that the device utilises, we used the following tools for synthetic and real-world benchmarking

• http://www.speedtest.net/
• http://www.ozspeedtest.com/
• IPerf network benchmark tool with Networkworld magazine’s test script

The Iperf settings to test TCP/IP throughput used were as follows:

(client) iperf -c <client IP address> -w 128k -i .5 -r -t 90 (server) iperf -s -w 128k

• File Upload and Download tests from local network and internet sources via FTP, 2MB Digital camera Photo JPEG file used, Windows command line FTP client. Either FileZilla FTP server for Windows or Unix/Linux FTP Server on the
• Latency tests using the PING tool

The average of three passes are taken and used, which we represent in our tabular results and charts.

When testing the 3G Internet connection, best effort was made to ensure that the internet connection was running in the highest possible speed band and signal strength, being 3.5G HSPA of theoretical maximum 7.2Mbit/s down and 2Mbit/s up, however at times the connection would fallback to slower 3G speeds of 3.6MBit/s down.

When testing the Wireless-G (Wi-Fi) Network connection, we used the default Wi-Fi Security settings in the Gateway being WPA-PSK Security using TKIP Encryption. This setting matches the default Wi-Fi Settings in Windows XP SP3.

Benchmarks

3G Internet performance benchmarks - Wired and Wireless Connection

This particular Gateway has been branded and configured by default to only operate on the Telstra Network [who are major Australian telecom provider] . The operation modes for best performance are 3G UMTS or 3.5G HSPA modes on either the 850MHz WCDMA band or the 2100MHz WCDMA band.

WCDMA 850 with HSPA offers the best performance, up to 5MBit/s of actual throughput out of 7.2Mbit/s theoretical maximum.

Our test unit seemed rather sensitive and due to the specifics of the units firmware the user is only able to forcibly select what frequency band they wish for the cellular modem in the Gateway to operate at, as such whether the unit operates in 3G or 3.5G is depending on signal strength, location and the weather, literally.

Signal Strength and the capability of the cell site to the user is critical for mobile broadband users to achieve good throughput and latency without dropouts.

We endeavoured to ensure that our 3G performance testing was performed under optimal conditions so we tested performance at several locations within the Sydney Metropolitan area and the Sydney Central Business District.

From our testing we recorded the best throughput possible with maximum signal strength and conditions as well as typical scenarios where medium strength and throughput is achieved with the possible 3G Modes and Frequency bands that the unit supports for a 3G Mobile broadband Connection.

The 2G/2.5G GPRS connection mode, which is to be used in locations where there is no 3G reception, was not performance tested.

Sydney Suburban location connected to Telstra ‘NextG’ Network in 3G UMTS mode on 850 MHz Band with full signal strength.

Throughput - FTP transfer

speedtest.net online throughput test

ozspeedtest.com Internet throughput test

Latency - Internet sites

Sydney Suburban location connected to shared 3-Telstra Network in 3.5G HSPA mode on 2100MHz band with full signal strength

Sydney Suburban location connected to Telstra ‘NextG’ Network in 3.5G HSPA mode on 850MHz band with full signal strength

Sydney CBD location connected to Telstra ‘NextG’ Network in 3.5G HSPA mode on 850MHz band with full signal strength

speedtest.net Internet performance test

ozspeedtest.com Internet performance test

Wireless-G (Wi-Fi) Local Area Network performance

Since this Gateway features only standard Wi-Fi functionality and is it advertised as a high peed wireless device, we conducted minimal Wi-Fi testing only to confirm that the performance is satisfactory. We tested at a range of five meters.

What do all these benchmarks mean?

3G Mobile Broadband, like Wi-Fi has large transmission overheads meaning that the advertised speed on the box, whether it is 7.2Mbps, 2Mbps, or 300Mbps are link speeds and not throughput speeds. For each real piece of data a user wants to transmit such as a document or photo, there is significant overhead on top of that file size due to the hardware and software needing to compensate for errors in the data stream through the air and having to retransmit that data as well as keeping track of information related to which access points or base stations the user is roaming between.

An analogy would be of a road that is sign posted with a 100KPH speed limit in rush hour but due to the congestion of all the cars trying to progress not one car is able to achieve the signposted speed due to the congestion and restrictions imposed upon each vehicle. Along with the 'box' and marketing specifications of 7.2mbps down and 2mbps up, Telstra do advertise real-world achievable speeds of "550kbps to 3mbps downlink with 6mbps bursts and uplink speeds of 300kbps to 1mbps with 1.3mbps bursts".

We easily achieved these 'real-world' speeds in several of our tests and exceeded them on several occasions, with the maximum downlink speed we achieved in the 400KB/s range with uploads in the 100KB/s range repeatedly and consistently provided the gateway connected in the optimal 3.5G HSPA mode. 3G yielded slower speeds as expected.

This speed is equivalent to a 4mbps/1mbps DSL or Cable broadband service, which is impressive considering it is using the public cellular network. We were able to replicate the promised speeds advocated by Telstra and of what we expected to see.

Early previews of 14mbps and 21mbps 3.5G HSPA products seen by NitroWare.net indicate that performance for connections which these link speeds are excess of 5mbps so this also affirms the results we achieved.

Unfortunately these maximum speeds are only possible if you like in an area [of Australia] with good 3G signal strength and is enabled for 3.5G HSPA and Telstra do advocate that the higher speeds are only available in specific areas such as Central Business Districts or Metropolitan areas or otherwise. Users should check whether the area they wish to use is 3.5G HSPA enabled.

While throughput can be very good, we found the latency to internet sites via Wired or Wi-Fi connection to be average which is to be expected for wireless networking. Some readers may see the latency results as poor but one must consider the factors at play, that wireless whether its WAN or LAN will always have high latencies, the distance of hundreds of metres or kilometres the cellular signal needs to travel, physical obstacles and the weather.

The mixed nature of 3G Internet is what leads to the inconsistency in some our results. We found it challenging to replicate the same result on a consistent basis especially when the signal strength(speed) or 3G mode changes on the fly