31 October 2009

Latency Lotto anyone?

I'll give you a hint, it's between 500-700ms first hop.

29 October 2009

Crickets.

2 days have passed since we logged the fault on our Bigpond service.

We've heard nothing.

"Have you tried turning it off and on again"

Yay Technical Support!

And so we start, Level 1 support. They go through their script. My partner explains the issue, tells the support droid that we've done the isolation tests, we've switched out modems, routers, etc etc. Explains that the problem has been confirmed by Telstra as congestion.

"It's a line fault"
"Really? So why is happening on my other line as well, which has an ADSL service with a different provider?"
"Oh."

"Well the line is obviously working now."
"I'm using VOIP on a different circuit."
"Oh."

And so it continued.

Eventually we're informed that if a tech has to attend the property, we may incur a service fee. My partner very politely tells them that this is a known issue that Telstra have admitted to *in writing*, and we will not be paying any service fee.

We're then told that it someone will get back to us in 2 days.

Here's something to ponder: If there is a 2 day process on reported faults, why do we need to ring when the issue is occuring?

It's now approximately 2 hours and 15 minutes since my husband first noticed the latency spike and called Tech Support.

My partner needs a scotch, or therapy. Possibly both.

Groundhog Day.

By this stage my partner is ready to go postal. Frustration levels are high.

He calls Telstra again in an attempt to sort out the authorisation issue. He's passed around again and eventually speaks to someone who checks the account and states that he cannot understand why Bigpond won't speak to my partner as he is most definitely listed as an authorised operator of the account.

Hurrah!

Notes are added to our file and he transfers my partner to Bigpond Activations.

The Bigpond Activations person read the notes on the file and advises my partner that the Bigpond account hadn't been configured correctly. No apology for the inconvenience caused, but we are past caring at this point, we just want to log a fault!

My partner gives the Activations person a brief overview of what has transpired in the previous call. He explains that he was the one that set up both the second phone line and the ADSL service. She tells him that she finds it highly unlikely that he was able to do that.

Yes, you read that correctly - Telstra accused my partner of being a liar.

By this stage, he is beyond frustrated.

Eventually the system is updated (as she apparently is one of only a few people that can see both systems - clearly she holds a position of power in the organisation) and transfers the call to Technical Support.

It's now approximately 6pm.

And we're back to where we were 2 hours ago.

Not with those shoes, pal.

On 26 October at approximately 4pm, latency hits 650ms (funny that this degradation of service is very close to school finish time. My, what a coincidence.)

Both the Internode service and Bigpond service are showing the same latency.

As instructed in the email from the @BigpondTeam, my partner places a call to Technical Support while the problem is happening.

I arrived home at approximately 5.15pm to find my partner still on the phone to Telstra, trying to log a fault. He was not a happy camper.

You see Telstra had told him he wasn't authorised to operate the account. Really?

He has *always* been authorised to operate the account. *ALWAYS*

If he was not authorised to operate the account, how did he manage to have the second line brought back into service and apply for Bigpond ADSL.

Just so we're perfectly clear:

It was my partner who arranged for the second line to be recommissioned.
It was my partner who arranged the Bigpond ADSL connection.

When he wanted to spend money, there was no problem with him signing up for all manner of services on an account that is in my name. Telstra/Bigpond could not have been more helpful.

Now there is a problem on that service, suddenly he's not authorised to operate the account. It's *his* name on the order number for ADSL, *his* email address as a contact.

In the first hour of this call, Bigpond rang his mobile to verify his identity while he was on the other line. He confirmed his identity with his name, address, date of birth, secret password on the account.

The Verdict? Not. Authorised.

It should be noted that the identity confirmation sequence he failed today, was the same identity confirmation sequence he passed when the line and ADSL service were ordered.

My partner asked to be transferred to a supervisor and after being passed up the food chain a couple of times, eventually landed at Bigpond billing who told him he was not authorised to operate the account.

He asked again to be transferred to supervisor and was told they would not escalate the call without first speaking to a person authorised to operate the account.

He gives up at this point.

1hr and 45 minutes later and we still haven't managed to log a fault with technical support.

Latency is 564ms, first hop.

Deja Vu... Vu.

And so here we are.

Same fault, different ISP.

I mention this latest development on Twitter and @BigpondTeam ask me to forward the details for them to investigate (props again to these guys - I hope Telstra realises the good these guys are doing with the Twitter user base).

Anyhoo, I send off the details and receive another prompt response from the @BigpondTeam:

You will need to lodge a complaint with Technical Support on 133 933 (24 x 7). They may send a technician to you or to the exchange. Either way, they will run detailed speed tests on the phone number associated with the BigPond account (I recommend calling at a time that you are noticing significant speed issues.) If calling is not suitable to your needs, they can be emailed directly below:

https://bigpond.custhelp.com/cgi-bin/bigpond.cfg/php/enduser/ask.php?p_type=technical

I have reviewed my previous email to you and it specifically states:

'It is unlikely that BigPond would supply you with an ADSL service on xx xxxx xxxx, due to the inability to provide you an acceptable service because of the issues at the RIM. What this means is that although ADSL works at your address, because of the issues with the RIM, BigPond would not put ADSL on at the address due to the poor service that you would receive. An order could be placed, but it would be highly likely that the mandatory further tests, that are done on all BigPond ADSL orders, would have stopped the order from progressing in to an ADSL service with BigPond.'

There were no guarantees made and my intention was based on the existing phone line. Each line has a different entry in to the exchange. However, I am surprised that an order has completed on any phone line from your property, but welcome you as a BigPond member.


Telstra are quite correct in that they were specifically referring to the number used for the Internode service. The Bigpond service is connected to a different number.

It's the last sentence in their response that I find interesting.

So we start the process again and my partner attempts to log a fault with Technical Support. And I do mean "attempts".

27 October 2009

Check.

Two weeks ago, my partner brought our old decommissioned phone line back into service and applied for Bigpond ADSL on 21 October.

On 26 October, our Bigpond ADSL service was activated.

My partner spoke to Bigpond and queried the availability of this service (given the information provided by Telstra previously that they would be unlikely to provide Bigpond ADSL to us because of the "issues on the RIM").

He was assured that there were no faults on any infrastructure. Really? Really Telstra?

How very interesting.

So here we are, a valued Telstra customer and the proud owners of two ADSL Services - one with Internode, one with Bigpond.

It should be noted that we have the links running on different equipment, totally independent of each other. Two very distinct and separate services.

So if the Telstra Wholesale reports are showing no spare ports on the DA we're connected to, where are we routing? Maybe we're on a different RIM? Could it be that by some magic or intervention by a supreme higher being, we have been transposed off the RIM completely? We should be so lucky.

It didn't take too long before we got to test both services. The usual nightly congestion started and we monitored both links.

Well will you look at that. Latency is woeful on both services. only with the Telstra service, we get 12% packet loss as well. Ace.

Request timed out.
Reply from 61.9.133.193: bytes=32 time=345ms TTL=123
Reply from 61.9.133.193: bytes=32 time=424ms TTL=123
Reply from 61.9.133.193: bytes=32 time=454ms TTL=123
Reply from 61.9.133.193: bytes=32 time=644ms TTL=123
Reply from 61.9.133.193: bytes=32 time=537ms TTL=123
Reply from 61.9.133.193: bytes=32 time=566ms TTL=123
Reply from 61.9.133.193: bytes=32 time=625ms TTL=123
Reply from 61.9.133.193: bytes=32 time=363ms TTL=123
Reply from 61.9.133.193: bytes=32 time=281ms TTL=123
Reply from 61.9.133.193: bytes=32 time=223ms TTL=123
Reply from 61.9.133.193: bytes=32 time=205ms TTL=123
Reply from 61.9.133.193: bytes=32 time=273ms TTL=123
Reply from 61.9.133.193: bytes=32 time=341ms TTL=123
Reply from 61.9.133.193: bytes=32 time=301ms TTL=123
Reply from 61.9.133.193: bytes=32 time=399ms TTL=123
Reply from 61.9.133.193: bytes=32 time=305ms TTL=123

Ping statistics for 61.9.133.193:
Packets: Sent = 103, Received = 90, Lost = 13 (12% loss),
Approximate round trip times in milli-seconds:
Minimum = 54ms, Maximum = 644ms, Average = 350ms

Congestion? Check.

Direct Relationship? Check.

Playing the Game.

It's a stalemate. Its fingerpointing all round. It's he said, she said, with a large dose of it's all too hard all rolled up in one tidy package.

Where to from here? The toothless tiger that is the TIO? They'll ping Internode as our ISP. No skin off Telstra's nose (even though the ball lies firmly in Telstra's court for resolution). Telstra Wholesale aren't a member of the TIO. Yippee.

My partner and I ponder our alternatives:

- Keep putting up with a crippled link, with both ourselves and Internode paying full price for a substandard wholesale service (as Telstra themselves admit)
- Do as Telstra suggested and move to wireless broadband (which does not meet our data or latency requirements), and is obscenely (eg $1000 a month) more expensive than our current solution and would require us signing up with Telstra on a 24 month contract. Over. My. Dead. Body.
- Move house.

Without the direct relationship with Telstra, we are in no man's land. No direct relationship so Telstra effectively give us a "talk to the hand" and fob us off back to Internode.

Fair enough then. If that's how you want to play it, we can play that way too.

SEP aka Someone Elses Problem.

So after I requested a further escalation point from Telstra, I received the following response:

I have spoken to several Telstra Departments today. I am sorry, but as you are not a BigPond Member, your ISP must be your escalation point. BigPond is unlikely to have connected you to ADSL on the phone line, due to the restrictions at the RIM. If your current service is not meeting your satisfaction, then I must refer you to your ISP.

Your current ISP must escalate any issues on your behalf. If any part of their service is not satisfactory to you, then you will need to communicate with them. I do understand your frustration and wish you well.


Did you catch that? You might have missed it. That was Telstra hiding behind the "you don't have a direct relationship with us" fairy. No, we don't. Telstra are correct.

However, the fault lies SOLELY with TELSTRA infrastructure. Internode have NO capacity, opportunity or ability to resolve this issue for us.

I particularly like the "if any part of THEIR service is not satisfactory to you". What complete bollox. Telstra KNOW that the issue is THEIR infrastructure, and NOTHING to do with Internode's portion of the service delivery.

Having said that, I was hoping Internode would have gone into bat for us just a little harder, instead of offering a templated hands in the air response like this:

I have looked through your fault tickets and I can see your service is affected by a congestion issue at your exchange. This requires our wholesaler to attend to a maintenance upgrade to there DSLAM at the Exchange.

I understand this is a frustrating time for you but unfortunately, we do not have any control relating to Telstra upgrading their infrastructure. We have had our Provisioning Investigations Team escalate this issue to Telstra, but we have not received and ETA yet for the upgrade.

As stated on our website and in our terms and conditions, Internode ADSL connections are not a service that we can provide an operational guarantee on, beyond the commitments made in our SFoA. We do guarantee that we will promptly investigate all matters brought to our attention and work with you to resolve any issues that we are able to; however, some things are simply outside of our control, such as those that occur within external networks and infrastructure and therefore we do not compensate for these issues.

We do apologise for this inconvenience and please be assured that we will be doing everything within our ability to keep escalating this issue to Telstra Wholesale.


So, just to summarise: Telstra say "not a bigpond customer, go talk to Internode", Internode say "nothing we can do, it's a Telstra problem".

And everyone pays full price up the food chain for a crippled service that works as intended a fraction of the time.

Wonderful.

Silly me.

On 3 October I submitted my whingefest to the Twitter feedback page.

I received a prompt response:

I have run extensive tests on the phone number that you are running ADSL on with another provider. There is no information to suggest that there will be any upgrades made to the capacity of the RIM. All enquiries for non BigPond Members, must be sent by your existing ISP on your behalf.

It is unlikely that BigPond would supply you with an ADSL service on XX XXXX XXXX, due to the inability to provide you an acceptable service because of the issues at the RIM. What this means is that although ADSL works at your address, because of the issues with the RIM, BigPond would not put ADSL on at the address due to the poor service that you would receive. An order could be placed, but it would be highly likely that the mandatory further tests, that are done on all BigPond ADSL orders, would have stopped the order from progressing in to an ADSL service with BigPond.

I would make the strongest recommendation possible that you think about moving to Wireless. I see that you have mentioned that wireless is not acceptable due to latency. However, we do offer antenna's that can be purchased to boost the signal and which should offer you a strong broadband service at faster speeds than you have mentioned that you are receiving via ADSL.


So, there's an admission that there's a problem. Sweet. Not going to ever get Bigpond ADSL provisioned on that link because of it. Check.

Wireless? Not an option.

I replied asking for a further escalation point. I also copied Internode on this correspondence, so they could see that we weren't waiting around for them to go into bat for us with Telstra - we were trying to achieve resolution ourselves as they seemed happy to take our money for what we have now had confirmed, is a deficient service.

I would have thought that Internode would have gone to their wholesale partner and asked for a reduced rate given the service both they (and us) are paying for, but not receiving.

Silly me.

Tweet Tweet.

About this time, I become a little more vocal in my online complaining about the congestion we're experiencing daily now. I play Latency Lotto with my followers on Twitter, where they try and guess our peak latency at a given point. Fellow RIM inhabitants sympathise and we compare notes. Gotta love the interwebz.

@Telstra and @BigpondTeam arrive on the Twitter scene. Hurrah! Another avenue of attack.

I ask for an escalation point for our issue from Telstra - given that we've had it confirmed by Telstra themselves, that it is indeed a congestion issue on Telstra infrastructure. Internode have thrown up their hands and pointed the finger back at Telstra and pretty much said nothing we can do - can't force them to upgrade capacity. Weaksauce.

And this is where I will give credit where credit is due - I have had more assistance and action from the @Telstra and @BigpondTeam Twitter guys than I have from Internode in the last 6 months and Telstra in the last 6 years.

But I digress.

@Telstra ask me to provide details of the issue via their Twitter feedback page.

I pretty much tell them the dealio since January 18 as you've read it here (albeit a little more professionally).

And this is where the story gets very interesting indeed.

Rinse and repeat.

By the 18th January 2009, we'd had enough. As our ISP, Internode were our first port of call.

We logged a call and Internode agreed to investigate and subsequently escalated the issue to Telstra Wholesale. We suspected congestion as the link was ridiculously slow at night, on the weekends and on school holidays. We submitted data to substantiate our findings.

By sheer chance, when the Telstra tech came out to test the link at the house, my partner was home. He spoke at length with the tech and explained all the tests we'd done ourselves and what we suspecting the problem was. The tech then toddled off back to the exchange. He rang my partner later that day and confirmed that our suspicions were in fact correct. It was congestion.

Fantastic! It was an identifiable problem that could be resolved!

Hurrah! Hur.. wah?

"Telstra have no ETA for resolution."

Ok, fair enough. We know they're busy making their billions of dollars worth of profit a year. Can't be expected to fix deficient services straight away.

So we wait. We contact Internode regularly (monthly) to ask for an update.

"Telstra have no ETA for resolution."

The Internode support ticket remains open.

Latency gets worse. Much worse. School holidays are a write-off. Easter we gave up trying to do anything. We told people if they wanted us to ring our mobiles as VOIP was unusable.

"Telstra have no ETA for resolution."

September school holidays? Unusable. Weekends? Unusable.

"Telstra have no ETA for resolution."

Merry Bloody Christmas.

We sucked up the 3mb cap. We still had ADSL right? It's still broadband, right?

Over the ensuing couple of years, we noticed the odd bit of flaky behaviour, but we still had ADSL right? We paid for our Internode "up to 8mb" service, and sucked up the fact that the RIM was capped at 3mb.

Last year, in an effort to extract ourselves a little further from Telstra's monopolistic clutch, we decommissioned our home phone line and opted for a VOIP solution with Internode. Woot, cheap calls! Three teenagers in the house can talk the leg off a donkey, VOIP was an awesome service.

Life is good. There is occasional connectivity flakiness, but nothing too bad. VOIP works, lots of donkeys sans legs around the place, but things are pretty peachy.

Timeline: November 2008. More flakiness starts appearing on the link. Sometimes the link is sloowwwwww. Like, painfully slow. Phone calls get choppy. Teenagers complain.

Christmas is a write-off. The link is butchered a lot of the time and is intermittently unusable.

Happy Frigging New Year.

Life is good. Or maybe not.

So finally, after 3 years of fighting and pleading and daydreaming about committing all manner of evil deeds against Telstra, we had ADSL. Glorious, fast, 1500/256 ADSL. It was only ADSL1, but hey, it was ADSL.

We decommissioned the 128K ISDN link and waved a fond farewell to the one-way satellite service (the dish still sits on our roof, gathering cobwebs, as a solemn reminder of those dark days).

At some point further down the track, Telstra upped ADSL capacity to 8mb. We jumped on the upgrade and bumped our Internode plan to 8mb/384. We were doing Mach 5 on the information superhighway!

For a year or so, things are great. Internode rocked our socks as a provider, the link was stable and fast. 8mb all the way baby, yeah!

But wait, what's this? Now we're getting 3mb throughput? Where did the glorious 8mb that we're paying for get to?

Investigations were made, questions were asked and it was confirmed that Telstra had capped the RIM at 3mb. Fan-freaking-tastic.

Watchdog to the rescue.

It should be noted at this point that Telstra had told us that for there to be ANY possibility of the RIM being upgraded, we would need 30 expressions of interest from residents in the estate for them to even consider an upgrade. We printed flyers and delivered them to mailboxes, we put notices up in the local area.

So, I came home from the doctor that day and started writing a letter to Watchdog detailing the painful mess to this point.

We were contacted by Dan Warne and he was very sympathetic to our cause. He spoke to me at length and then to my partner about the more technical aspects of our journey so far.

Our letter appeared in the APC Magazine in November 2004 and Telstra were asked for a "please explain".

On 10 December 2004, a sync light appeared on our modem.

A little history to bring you up to speed.

In 2002, we moved into a new house in a new estate in outer south eastern Melbourne. We had checked prior to moving in that we could get ADSL. We dutifuly put our address into the "Can I get broadband?" website and were pleased to find that yes, we could.

We move in and applied for ADSL. And there started a long, painful and incredibly frustrating journey that continues today.

As it turns out, the "Can I get broadband?" website fails to take into consideration any type of technology blocker to getting ADSL (eg pair gain, RIMs etc). So yes, the Berwick exchange was ADSL-enabled - the RIM we were stuck behind, was not.

So we went from uncapped cable in our old house, to dialup. For people that worked in IT and had a requirement for remote connectivity, we were not happy.

To cut a long story short, the next 2 years saw us fight to have the RIM enabled. While trying to get ADSL into the estate, we were forced to use a one-way satellite service, and later 128K ISDN in order to get the connectivity we required. I had a legacy Telstra direct service that I'd had since 1997 which we used for the satellite uplink.

The Satellite connection had a 3gb per month limit and excess traffic was charged at 25c/mb. Interestingly, while classed as "broadband", this satellite service was excluded from all the 'free sites' that other broadband services enjoyed. Thanks Telstra!

We struggled with one-way sat for a couple of years before we bit the bullet and had 128K Onramp service provisioned. I think it was called Home Highway or something.

At this point we were paying close to $650 a month for "decent" connectivity.

During this time, we had been constantly ringing Telstra and registering our "expression of interest for ADSL" on the ADSL register. Every month we would add our name to the register, and every month would check how many expressions of interest were on the register, only to be told "none". We would go through the process again, add our name to the register again (which would "fall off" the register every month), and rinse and repeat.

We continued applying for ADSL with Internode and the answer was always the same: "no alternative path found". In approximately August 2004, success! (albeit short-lived). Our application was now sitting at "provisioned". Children danced in the street, crowds rejoiced, baby birds twittered. Oh.. what's that? It's Clayton's provisioned? Where "provisioned" means Telstra in their infinite wisdom, transferred us to another RIM... with no spare ports. Great.

So we've gone from first in the queue on a RIM that may be upgraded, to last in the queue on a RIM that is already at capacity. Hurrah! Numerous phone calls to Telstra regarding the transposition and requests for even the tiniest bit of information was met with silence.

One day I was sitting in the local doctor's surgery reading a copy of APC magazine. And I came across the watchdog section. It was a lightbulb moment.

Welcome to hell.

Hi and Welcome to the chronicles of one family's dealings with the monopolistic behemoth that is Telstra.

Enjoy.