|
The company I worked for was a German subsidiary of a Norwegian company named EDB, which is an abbreviation of “Electronikse Data Behandling”. EDB is an IT service provider for various customers ranging from public institutions, industry customers to financial companies. We provided all services including pre-configured PCs, servers and the wide area and the local networks.
In this company I belonged to the “Network Services” group, which was mainly based in Norwegian sites; I was one of two members based in Europe and I became a so called ROR ( Regional Operation Responsible ) for the area Germany, Austria and Switzerland and some other sites in Eastern Europe, which belong in an organizational perspective to sites in the countries named.
Following some generalities and corporate standards provided by the group I independently operated the networks of all customers in my area. I covered almost every kind of work you can imagine. Beginning with the design of an entire new network or just an extension requested and the creation of proposals, which contain of course the charge for the devices to be installed, but also estimations for travel costs and other efforts for e.g. procurement, special configurations, installation work times. If the order has been placed, planning comes next. I always plan twice and compare to have the correct equipment delivered. So as my orders were very precise even including the style of the power plug for the country a device was intended for, I never struggled with wrong deliveries.
Due to my international focus I also had to face spectrometer analyses in other languages like Spanish or French as well. The customer mostly ordered this just from a local company or still had the initial measurements still in the drawer. Nevertheless I always managed to figure out, what kind of fiber (distance, damping etc.) I have to deal with and therefor was able to order the appropriate modules ( SFP/GBIC ).
Fortunately I didn't have to mind the purchase itself. This was handled by a central department of the company. I sent a mail containing all the devices, their special types numbers, needed modules and cables and after 4 weeks the devices were either sent to my attention or right to the customer's site in case of bigger projects. In those cases or tricky, complex installations or if it came to projects in production environments, where the maintenance window was very tight, the schedule had to be kept at all costs and functionality was a must, I traveled to the site personally.
I pre-configured, the devices, installed resp. replaced others. These also could have been 3rd party ( non-Cisco ) devices, where I had to read out the given configuration and map the function to the Cisco device by applying an adjusted configuration. I verified the functionality by conducting of some tests with the people involved and received the acceptance from the customers representative. In case of small extension e. g. if a customer would just soon run out of network ports in a specific place, I pre-configured the device needed and sent to a local contact, who installed the device physically and connected it to the given network according to my advice. Then I was able to adjust the configuration according of any special needs. Afterwards the local contact arranged a test with the customer.
Of course all installations and changes had to be documented; there was a central document management platform provided. In the first years we used a Lotus Notes Database, later SharePoint was used.
Operation is the next matter. It includes troubleshooting, monitoring and maintenance. Working on problems could be anything starting with simple interface problems, Duplex problems or other obstacles which lead to interface errors and these in turn to performance problems. Various network cards of different brands all prefer their own setting on the switch even if you like to have the setting the same on all ports, which would free you up from adjusting as soon as a node is moved to another place and then connected to another port.
The last line of defense in troubleshooting is Wireshark. But I remember only 1 time, that I needed it to find a Cisco derived problem, which was one in OSPF to find the reason why a backup line was not used, when the primary line was cut off. But this was a long time ago, when Wireshark still was called Ethereal and Big Ben was a little watch ( and we used the original sniffer from NetScout instead ). In all other cases I sniffed the network to find PC problems ( PCs of a whole site froze for seconds ) or server problems ( access to applications failed, client / server or database systems communication problems etc. )
Customer requests like implementing VLANs and spread them all over the network, routing between VLANs, also with filter function on the the router once in a while are welcomed variety to the daily work as done pretty easy. Rather unusual were the implementation of MAC address filters or the configuration of multicast groups on a switch. It gets tricky, if the customer would like to move an application server behind a firewall, but cannot change the IP address due to license reasons of the application running on that server. Interesting projects develop, if a customer buys another production area right in the neighborhood and asks for an integration of both networks into one. This requires major infrastructure changes, while the use of the default VLAN in both networks is one of the lighter issues.
A couple of times I also was asked to install servers the colleagues from the appropriate group provided as this comes often with the configuration of special features like the creation of etherchannels using 2 or more interfaces to increase availability and the throughput to and from the server. The management ports of the servers also required special configuration. I implemented the monitoring of these special ports involved on the switch and also surveillance of some essential parameters on the server like the availability of background applications and hard disk space. I also tried to write a script, which monitors the usage of the server memory, but due to the fact, that this is working quite reliably today (even) on Microsoft server I didn't have the chance to catch much experience, if the script could act as a trustworthy function. Any alarm triggered by the script about high usage showed only a temporary problem and it's cause could not be evaluated later on.
All this I didn’t do “just” for my area, I also supported other sites in other countries like Italy, France and Spain, when necessary including whole network replacements. In the later case I traveled to the site, made an assessment and wrote a report with a recommendation for the replacement and a rough estimation of the costs. While site managers in Germany are very strict and carefully and those replacements only could be done e. g. between Christmas and New Year’s Eve, when the site is down anyway, I was allowed to do this work even during production shifts in the other southern countries named. Nevertheless no site ever claimed, that I disturbed the production; only the directly involved people recognized the change, but the management was surprised, when I came up saying, that I was done.
Besides my activities in the pure LAN area I also was member of the WAN group. Inhere I also installed other kinds of devices like firewalls ( Nokia with Check Point FireWall-1 software ) and caching / data compression devices ( Riverbed ). I had access to the firewall as far as to the rule base and all related monitoring / troubleshooting functions. So I was easily able to check, which node is talking to which other and may be see, that the rules were not configured correctly. It was agreed, that only a few people in the company should create and adjust the rule base, since this is a tremendous security issue it has has to be maintained closely and and this group created special databases to keep overview and verified that all the rules followed the security guidelines agreed with the customer. But the colleagues were all happy, when the problem ticket came from me, it contained precisely on which firewall in which rule something has to be added or adjusted. Thanks to that I got my rule base changes needed within hours. In urgent cases I just called one of the guys and then it was a matter of minutes. Generally I think I can tell, that I got along pretty good not only with these colleagues but with all others as well. I was respected also from my managers, because of my knowledge and the reliability I did my work with. I have served the customers self dependent and self-managed and I was also a direct contact point for the IT and site managers, who were highly satisfied with my service delivery, responsiveness, my unconditioned readiness (to go out even in the middle of the night to replace a broken device) and qualified consultancy.
The WAN connections between the sites and central nodes were mainly done by IPSec tunnels. I configured VPN concentrators (initially Nortel later we began to use Cisco ASA Boxes) and cared for the installation with a test e. g. of the throughput afterwards (iperf). The operation of Name and DHCP services were also assigned to the WAN group. ( Well, Microsoft needs it's own name service to work, but that's another story ). Maintenance in DHCP ranged from just entering new nodes with static IP addresses, to the entry of special values in scope options needed by applications up to the creation of a new design for a whole network in preparation to a move to another IP address range mostly because the given one became too small, hence we ran out of free addresses. A tool I wrote to discover "living" IP addresses brought some surprises, if the customer secretly installed devices without letting us know. Fine art starts, if you can do this for remote networks and still are able to see devices without a default gateway entry (without any other help I mean). We preferred to move the devices to prevent huge local routing, which easily can lead to performance problems.
The latest rollout was to replace the WAN by connections to a MPLS network of a new provider. I was involved to migrate the WAN worldwide e. g. in the US, India and South America and connect the LANs of the respective sites to it.
To monitor all the equipment I used Nagios based on Linux, which I setup and operated on my own. It is very stable and reliable, highly configurable. It reliably informed me by SMS, when a node went down. Some Linux scripts I wrote for e. g. saving the routers’ and switches’ configurations made my life easier. Later on all the devices have also been included into CiscoWorks for Switched Internetworks and Tivoli Netview, which were run centrally on hosts in Norway, but I always kept "my" Nagios, since I also monitored special local setups. Some other helpful scripts completed my toolbox, e. g. to mention a function, which reads the time of the last change of the operational status of a port on a given switch. This is very useful, if it comes to replacements and consolidations. Another one discovered hubs, some users install secretly to avoid extra costs, but were not allowed according to the customer's own regulations. Or one, which monitors changes to routing tables or tunnel interfaces to discover outages of internet access lines behind the VPN router. So if I needed anything no solution was available for I just sat down for an hour, wrote a small program and that way I got exactly what was required.
Little things in between which also made my day varied were some repair works like replacement of CPU fans, abuse of an old PC's power supply for a VPN controller, which original power supply broke and a new on wasn't fast enough to get. Until we had a local server guy I was also contact for all server related issues. So I replaced hard disks or brought servers back to live and made them basically available for the colleagues in case one crashed. I wrote a windows tools to verify the home users installation for the capability to create a user VPN out of his environment. Rarely I soldered interface cables for customer provided modems or tailored RJ45-plugs to cables.
Special actions to mention:
Rebuild of a customer's complete server room. All devices were either build in stands or just placed on wooden tables. It took 26 hours of continuous work - only broken by waiting times generated by another company, which spliced the fibers into the new cabinets. But this wasn't the only round the clock work I did, but I shouldn't tell too loud, since this easily goes close to the gray zone of the German “Arbeitsrecht” ( Working rights ). From my point of view the problem is not, that it could happen anything at work due to a may be reduced attention ( what I never had to fight with ), but driving home safely afterwards.
Besides :
Project manager for whole site migrations, locations which had to be integrated into a given customers network as they have been acquired. After all assessments has been made I acted as the central contact for all means of the migration. I interfaced between the customer and our internal groups e. g. the server guys, who had to set up a new server, takeover all the data from the given one and finally I agreed with the customer the time and date for the big bang. I organized that the customer procured all the needed software and agreed to an update scheme not interfering with his production needs. Some helping hands I had to direct re-installed all the PCs based on a common Windows version with a software distribution on top. Besides all the organizational work it was of course my job to update the customers network and equipment and document the new site. Often I had to replace the whole network, if I found outdated or leased / rented equipment. There were also the cases, when we couldn't replace any customer provided firewalls and WAN links at once, so I just took over control and maintained the 3rd party devices until the wan links became redundant as replaced by a connection via corporate outbreaks.
Well, this is already I quite long list, but I’m sure it does not contain everything I did.
|
