Lte Operations and Maintenance

LTE OPERATIONS AND MAINTENANCE JAKARTA SEPT 10th 2013 Lawrence Williams 9/11/2013

For internal use Document ID / v. 0.1 / Life cycle status / Dept. / Author

©2013 Nokia Solutions and Networks. All rights reserved.

CONTENTS • • • • • • • • • • •

Introduction O&M Architecture/NetAct IOMS BTSSM LTE Configuration and SW management SWAM (Licences) ALARM Handling Process Performance Management Troubleshooting LTE Tools SON For internal use

2


INTRODUCTION • LTE Should Revolutionise O&M in Mobile Networks • Self Organising Networks (SON) Specified in 3GPP REL 8 36.902 • In Reality Early SON implementations have proved unPredictable (ANR) or non trivial to manage (eNB Autconfig) • SON will become more reliable and understood and More SON Features will be released Step by Step • Currently Most LTE O&M operations are using familiar apps from 2/3G • Big Changes in O&M apps will occur when Upgrade from OSS 5.4 to NetAct 7 For internal use

3


MAIN DIFFERENCES IN LTE O&M TASKS FROM WCDMA • • • • • • • • •

No Object Browser In APPLAU (Use TLUI Status or SiteBrowser) iOMS located in NetAct Rack . Responsibility of OSS Engineer? Many New Parameters and Counters particular to the technology As no RNC in middle it may be harder to trace Transmission IP issues eTe As relatively New Technology Must be aware more device compliance issues Licence Management ( Use of SWAM) Challenges to O&M to understand SON automated tasks eg auto ANR ,PCI etc Most important to Maintain Voice cabability (CSFB,SRVCC) Greater Mobility Issues to Consider LTE->3G->GSM

4 9/11/2013



OAM Architecture key tool for centralized SON

From the users’ point of view, managing the Nokia Networks Mobile LTE/EPC network is very similar to that of Nokia WCDMA when NetAct is used. Management system traffic to and from LTE/EPC network elements always goes through NetAct. The interface between the Flexi Multiradio Base Station and NetAct (iOMS) is based on the BTSOM protocol. It carries all the necessary data and commands (for example, alarms, measurements, configuration and new software data) to control the network element behavior remotely. For internal use

5


iOMS in NetAct site

• The LTE iOMS is a part of the NetAct concept. Although iOMS is running in its own blade unit, it can be located in the same rack as the NetAct Blade System

For internal use

6


iOMS SW management iOMS EMs • iOMS SW can be installed either by

• Using FEWS • Using USB stick • iOMS has only following EMs supported:

• Application Launcher (Java): • FM GUI • Active Sessions • Parameter Tool • PUTTY • Also CLIs provided by FP and OMS are supported. For internal use are disabled (e.g. Topology Browser) • All other EMs ©2013 Nokia Solutions and Networks. All rights reserved.

7

O&M TASKS STATUS MONITORING B. ALARM MONITORING C. PARAMATER OPTIMISATION D. SW ROLLOUT/ROLLBACK E. CONFIGURATION BACK UP F. LICENCE MANAGEMENT G.PERFORMANCE MONITORING H. TROUBLESHOOTING A.

8 9/11/2013



BTS Site Manager Application • The content in the Task view changes according to the selection in the View bar

Task view

Properties view Equipment view

View bar

Alarms from the eNodeB and its transport module

Fault view

For internal use

9


• Example: BTS Hardware task view consists of Equipment view (shows the BTS cabinet with units/modules and cells)

and Properties view

BTS Site Manager Fault Monitoring

BTS Site Manager shows the faults coming from the site Fault view shows the list of faults Faults can be • Warning • Minor • Major • Critical Fault descriptions available in online Help

For internal use

10

Shows/Hides Fault history of current manager session

All Active Alarms

Opens Block toggling ©2013 Nokia Solutions and Networks. All rights reserved. faults

Additional info about the selected fault Show Fault source in EM equipment view

BTS Site Manager Fault Monitoring

You can show faults for an alarm type or fault history

For internal use

11


Introduction BTS Site Browser Provides access to multiple predefined BTS sites simultaneously Speed-ups and helps BTS Site management especially In trials and rollouts There is no NetAct available New BTS Site SW version is not supported by NetAct

Uses standard BTS Site Manager interface towards BTS

Trial and Rollout Activities

BTS O&M SW version not supported by NetAct

NetAct not in place

1st Mile

Core

Controller/Gateway Sites

For internal use

12


Deployment

• Installed with BTS Site Manager – BTS Site Browser is installed to BTS Site Manager installation folder Default location •Windows: C:\Program Files\Nokia Siemens Networks\Managers\BTS Site\BTS Site Manager\ •Linux: opt\Nokia Siemens Networks\Managers\BTS Site\BTS Site Manager\ Can be started with BTSSiteBrowser.bat or BTSSiteBrowser.sh from BTS Site Manager installation root folder

• Define properties – You have to create text file where is listed all IP addresses of BTS Sites managed by BTS Site Browser – File -> Properties Define text file where IP addresses are listed Define installation location of BTS Site Manager, so it can be started from BTS Site Browser

• You can alsousegive address list as http address for example from OMS For internal 13


https:///BtsAddressList.txt

BTS Site Browser main view

List of predefined BTS sites

Information about selected site

Block/unblock cells. More info…

List of faults from successfully scanned BTS sites For internal use

14


For internal use

15


FAULT MANAGEMENT

For internal use

16


Fault Management Architecture

NWI3 Interface

Active alarms

Alarm history eNBs/iOMS alarms

For internal use

17


Fault Management NetAct Monitor application

• All relevant Alarm Monitoring • Rule Editor tools within a • Rule Explorer Common • Object Explorer • Managed Object List Desktop. • Object Search • Properties • Possibility to • Filter launch Explorerall • Filter Editor OSS5.1 • Quick Search • Network View applications • Network View Editor and element • Notes managers • Trouble Ticket List from • Trouble Ticket Details the desktop • Alarm List • Alarm History • Alarm Details • Warning List

View Explorer

Alarm List

Object Explorer Properties and Notes

Network View

For internal use

18

©2013 Nokia Solutions and Networks. All rights reserved. Introduction to NetAct / Nagavishnu Nagaraj

Alarm Details

FM IOMS GUI

For internal use

19

1

©2013 Nokia Solutions and Networks. All rights reserved. 2011-02-03

NetAct Application (NetAct OSS5.3 CD Set 1)

For internal use

20


Monitoring

• All relevant Alarm Monitoring tools within a Common Desktop • Possibility to launch all OSS5.x applications and element managers from the desktop For internal use

21


Alarm Monitoring

For internal use

22


Alarm Monitoring Application

Active Alarm List grouped by

Alarm

Alarm class

Details

Alarm Blinks when not acknowledged

Access to Alarm Manual

For internal use

23


Alarm History

For internal use

24


Alarm History

For internal use

25


Fault Management Features PM – FM Interworking in BTS LTE166, [RL20] 2011 - The performance values of specific KPIs in the Flexi Multiradio BTS are supervised. • The set of supervised KPIs is predefined and fixed in Flexi Multiradio BTS. • It does include few (around five) important BTS related KPIs • The KPI values calculated in Flexi Multiradio BTS will be compared to defined threshold values. • Overall constraints will be taken into account as e.g. sufficient traffic ongoing to be able to evaluate the situation. • Default threshold values are pre-defined in the Flexi Multiradio BTS, but can be changed by operator They are changeable as parameters in the configuration database.

- If the KPI value does exceed a defined threshold value, an alarm will be raised. - If the value drops under the threshold value the raised alarm will be cleared again 26

- If the For value drops under the threshold value the raised alarm will be cleared internal use again. ©2013 Nokia Solutions and Networks. All rights reserved.

CONFIGURATION MANAGEMENT PARAMETERS

PLANS

BACK UPS

UPLOADS

AUDITS

For internal use

27


Configurator: Application for CM Planning Center / 3rd party system The major components of Configurator • CM Editor

Configurator

– Display actual parameter values – Create, modify and delete plans

Northbound interfaces

• CM Operations Manager – Upload, download from / to the network – Import, export via northbound interface

• CM Analyzer – Run consistency checks – Define user-specific consistency checks

CM*

Radio/Core Siemens GSM & WCDMA

Nokia GSM

WCDMA

WiMAX & I-HSPA & LTE

Core Rel4

For internal use

28

CM Analyzer

Southbound interfaces

• Southbound interfaces • Northbound interfaces

CM Operations Manager

CM Editor


Multi-vendor integration

Basic Operations Customer’s planning system

Export Element Manager

Import

Regional NetAct launched Configurator CM events Upload

CM Provision • download • activate

Local operations For internal use

29


Network On the field

OSS5.1

CM Essentials - CM Editor Description • Editing & browsing through actual and planned configuration data • Integrated plan management – For all network elements (BSC, RNC, MSS, MGW etc.) – Parameter mass modification – Advanced search – Template management Benefits • Operator can compare between actual and planned parameters while editing • Find objects that match specified criteria with possibility to set new values only to these objects For internal use • Templates to complement planning tool 30 data for minimizing user errors


Data Management - CM Operations Manager Description • Keep Configurator up-to-date with network – Upload actual configuration

• Efficient plan-based management – Plan provisioning to network – Import plans from Nokia Siemens Networks or 3rd party planning environment – Export actual data and plans – Compare actual with planned data

• Benefits Scheduling of operations

• O&M Operations Manager – all operations under control For internal use • Comprehensive feedback to all 31 operations


OSS5.1

CM Analyzer Description

• Consistency checking for – Planned and actual configurations – Parameter values – Dependencies within a network element – Dependencies between network elements Benefits

• Scheduled start of consistency checks • Automatic corrections of inconsistencies • Automated reporting that can be sent to e-mail address • Detection of planning errors including For internal human errors atusean early stage 32 • Higher quality of network plans


For internal use

33


Major functional changes between NetAct 5 and NetAct 8

NetAct License Manager

Monitoring as the alternative for license key management

NetAct 8: • Nokia Siemens Networks’ software asset monitoring via SWAM client as an alternative for license key management. • Supported use models: • Pay as you grow • Pay as you configure • Pay as you use

NetAct 8 CM, PM database

NSN SWAM Client

Operator

For internal use

34


SWAM Technical description SW feature activation parameters

NetAct NSN NetAct CM Database

SWAM Client

Monthly reports

Operator

Radio Network

For internal use

35

• Operator configures RAN parameters as per internal SW license investment plan • NetAct configures the Radio Network as per the input parameters • SWAM client runs the report query at predefined intervals and creates reports of configured/used parameters • Reports are sent to NSN and operator at predefined interval • Data isSolutions archived ©2013 Nokia and Networks. All rights reserved.

LTE Basic SW – Pre-Paid model • Operator configures the number of Connected UE for each eNB (value equal to the purchased license for BSW) • The maximum number of Connected UEs per eNB for Flexi Multiradio BTS is controlled by a configuration parameter “maxNumActUE” • Connected UEs exceeding the configured value of the parameter are rejected • SWAM client gathers the value of this parameter for each eNB periodically andNetAct CT uses SWAM report to ensure the adherence to the Highest peak value allowed contractual agreement CU Configured value 75 of Connected UEs Rural

10 15

suburban

eNB

20 30 50 75

time

At any given time 76th user will be rejected by scheduler

Dense Urban

For internal use

36


LTE Basic SW – Post-Paid model • The maximum number of Connected UEs per eNB for Flexi Multiradio BTS is stored by PM counter (ENB_LOAD_ACT_UE_MAX) in NetAct • SWAM client gathers the value of this PM counter for each eNB periodically and compares it to previously stored highest value • If there is a positive change in the value, the delta value is reported and CT will invoice the customer for this delta value per eNB • The past highest value is replaced by the current highest value and stored in the database for the future report comparison Counter value of Connected UEs during measurement period Rural

10 15

suburban

NetAct

Max.Connected Users

eNB

20 30 50 75

Dense Urban For internal use

37

Counter

Period current 1

Period current

72

Value 75 is stored as new Highest value to be compared Solutions and Networks. All rights reserved. for©2013 theNokia next report

75

Invoicable value

3

PERFORMANCE MANAGEMENT

For internal use

38


3. LTE RL40 measurements LTE RL40 measurements are divided into two subgroups as presented bellow: RNW Measurements

RNW Measurements

LTE S1AP (M8000)

LTE Inter System Handover to GSM per Neighbour Cell (M8019)

LTE Cell Load (M8001)

LTE Cell Availability (M8020)

LTE Transport Load (M8004)

LTE Handover (M8021)

LTE Power and Quality UL (M8005)

LTE X2AP (M8022)

LTE EPS Bearer (M8006)

LTE UE and Service Differentiation (M8023)

LTE Radio Bearer (M8007) LTE RRC (M8008) LTE Intra eNB Handover (M8009) LTE Power and Quality DL (M8010)

Transmission Measurements LTE IP Statistics (M51120)

LTE Cell Resource (M8011)

LTE PHB Statistics (M51121)

LTE Cell Throughput (M8012)

LTE Ethernet Link (M51123)

LTE UE State (M8013)

LTE IP Sec (M51125)

LTE Inter eNB Handover (M8014)

LTE IP Filtering (M51126)

LTE Neighbour cell related Handover (M8015)

LTE VLAN IP Statistics (M51127)

LTE Inter System Handover (M8016)

LTE VLAN PHB Statistics (M51128) LTE VLAN Statistics (M51129)

LTE Inter System Handover to UTRAN per Neighbour Cell (M8017)

LTE ETH_IF Statistics (M51130) 39

LTE eNB Load (M8018) For internal use /


4. Pre-defined reports overview -> System • System Program gives a general overview of network status. With this report the user can achieve in high level how network is performing and understand if problems are occurring. This report should be used to benchmark the network.

40

For internal use /

Program These reports include NSN LTE RL30 benchmark KPI's, such as: • Cell Availability • RRC Connections • Radio Bearer • E-RAB • Data Volumes and Throughputs • HOs and mobility analysis • RLC Retransmissions and HARQ • Active Ues • Quality


For internal use

41


4. Pre-defined

reports overview -> Accessibility and Retainability

• Accessibility and Retainability subset provides detailed analysis and root failure causes for problems when trying to access the network or troubleshooting of drop calls/services. These reports should be used to understand why calls are being rejected or dropped. • The subset is divided into four reports: • Accessibility and Retainability • Network Access - General and S1AP • Network Access - Service Access and RRC • Service Retainability

42

For internal use /

These reports include metrics for the following issues: • Attach and Setup SR • Initial Context Setup • S1 interface • Paging • E-RAB access fails per cause • RACH stp SR • RRC Access fails per cause • ECM idle to ECM Conn transitions • RRC releases per cause • Data RB releases per cause • E-RAB normal releases • E-RAB drops per cause • RB Retainability


4. Pre-defined reports overview -> Mobility • Mobility subset provides handover and mobility statistics. Decision rate, These reports include Intra eNB, Inter eNB and Neighbour HO performance are available. metrics for the following These reports should be used in the optimization of neighbors lists. It issues for intra and inter HO: should also be used to analyze the performance of HO analysis for • HO Decision Rate UTRAN and Network Assisted Cell Change for GSM per neighbor.

• HO SR • HO attempts • HO preparation SR • HO Prep FR per cause • HO Drop Ratio per cause • HO Preparation, attempts and fails per Neighbour cell • CS Fallback • NACC • UEs in DRX

43

For internal use /


4. Pre-defined reports overview -> Interface •

Interface Analysis subset provides statistics of X2, IP and Ethernet Interfaces. Metrics per VLAN and per QoS Class are also available in these reports. These reports should be used to analyze if throughputs and data volumes are the expected ones in X2, IP and Ethernet.

analysis

This Subset is divided in six reports: •X2-Interface Control Plane data (incoming/outgoing throughput and data volume) User Plane data (incoming/outgoing throughput and data volume)

•IP interface traffic Incoming/Outgoing volume Incoming/Outgoing dropped congestion

•IP interface traffic and QoS classes PHB Incoming/Outgoing data volume and throughput per QoS Class

•Ethernet Interface Inbound/Outbound data volume/throughput Received Ethernet frames SR

•VLAN analysis per QoS class Inbound/Outbound data volume and throughput per VLAN and per QoS class 44

For internal use /

•VLAN total volume analysis


Total IP data volume per VLAN

4. Pre-defined reports overview -> Modulation and Coding Scheme • Modulation and coding scheme subset provides This Subset is divided in six reports: statistics of data volumes, transmissions, share of transmissions and successful transmissions per MCS. This reports should be used to understand the quality of data volumes transmissions and DL Adaptive Modulation and Coding (AMC) algorithm performance to improve system capacity and peak data rate.

• Transmissions and Volume per Low/High MCS PDSCH and PUSCH transmissions using low/high MCS Percentage of DL/UL traffic volume using Low/High MCS codes

• Percentage of transmissions per MCS PDSCH and PUSCH

• PUSCH 1ST transmission per MCS • Unsuccessful Transmissions per MCS • NACK MCS time division •Data Volume per MCS MAC PDU volume on PDSCH/PUSCH per MSC Share of DL/UL Traffic per MCS 45

For internal use /


4. Pre-defined reports overview -> Integrity This Subset is divided in six reports:

• With Integrity subset is possible to analyze delays, • IP Packet integrity congestions and encryption/decryption metrics. These ESP packets encryption SR reports should be used to understand delays in Dropped packets due to ingress rate limiting different protocols levels. Discarded packets due to Filtering violations

• Packet delay PDCP SDU delay UL/DL (min, max and avg) DL/UL SCH TB Retransmissions and error ratio RLC PDU Retransmissions

• BLER per Modulation QPSK, 16 QAM Gain using HARQ

•Congestion per QoS classes for IP PHB IP PHB Packets to send drop rate and data volume lost due to congestion per QoS class

•Congestion per QoS classes per VLAN IP VLAN PHB Packets to send drop rate and data volume lost due to congestion per QoS class 46

For internal use /

• IPsec packets


ESP bypassed (ingress/egress)

4. Pre-defined reports overview -> Usage • With Usage subset is possible to analyze throughputs, resources utilization, power and number of active users. • With these reports, throughputs at different levels should be checked to see if it matches the network configuration. They are also very important to understand the Physical Resource Blocks utilization for issues like network upgrades, power analysis with downgrades/upgrades in scheduler and MIMO performance. They are also very important to understand the throughputs per Quality of Service Indicator to check the distribution of network resources per group of users priority.

47

For internal use /

This Subset is divided in seven reports: • Throughput at PDCP level and per QoS class

PDCP UL/DL thp (min, max and avg) PDCP SDU THP and data volumes QCI • Physical Resource Block utilization PRB utilization UL/DL DL/UL PRB Analysis per TTI, per UE DL/UL PRB utilization level per TTI • RLC, SDU, RRC, analysis RLC SDU data volume per channel MAC SDU volume per channel RRC UL/DL Traffic volume • Power UE Power HeadRoom for PUSCH per level UE Power analysis (scheduler upgrades/downgrades) UE Delta Tx PUSCH/PUCCH per level • MIMO Open Loop Diversity/Spatial Multiplexing Mode Usage and Switches ©2013 Nokia Solutions and Networks. All rights reserved. • Active users per eNB and per cell

TROUBLESHOOTING

48

For internal use /


PROBLEMS ENCOUNTERED IN LIVE NETWORK LAST 1.5 YEARS RL20+ • • • • • • • • • • • •

RP3 ALARMS AUTO CONFIGURATION ANR CREATE TOO MANY ADJ PCI MANUAL PLANNING MISTAKES NETWORK OUTAGE CAUSED BY MME PARAMETER CHANGE NETWORK OUTAGE CAUSED BY SIG LOAD ON S1 DUE eNB MASS SW Ug SLEEPING CELLS (Related to RP3 Delay) CONFIGURATION CHANGE AFTER SW Upgrade UE SW Bugs THROUGHPUT ISSUES ,FRAGMENTATION FTIB LOST VOICE CAPABILITY (CSFB Issues) RACH STORMS (eNB SW Bug) For internal use

49


Troubleshooting guidelines – Problem type according to KPI classification Problem type according to main KPI categories: - Accessibility problems: • Service accessibility, e.g. Attach, or EPS Bearer Setup Request success rates - Reliability problems: • Once established, the reliability of the service, e.g Voip call, or EPS bearer drop rates - Integrity/Error rates problems: • Promised to deliver the service with a certain quality. E.g. Cell throughput, or user data rates, packet delay. - Availability problems: • Relationship between service up and down times, e.g. cell, or LTE Network availability - Mobility problems: • Different mobility related metrics, e.g. hand-over success or drop rates, For internal use ©2013 Nokia Solutions and Networks. All rights reserved. 50 latencies.

Troubleshooting guidelines - The above-mentioned cases it is essential to try to get at least following logs : • eNB snapshot • Interface traces S1-C by wireshark - The eNB is connected to the EPC (S1 interface) by IP switches. The precondition for capturing traces on these interfaces is the availability of a mirror port on one of the switches. The tracing on these interfaces can done either using wireshark (usually for lab tests) or using commercial protocol trace equipment (e.g. NetHawk M5, JDSU SART). If the IP traffic on these interfaces is encrypted (e.g. if IPSec is used) it is necessary to use the deciphering functionality of the protocol tracers. • Emil logs - BTS message remote tracing: BTS message remote tracing can be configured, started and stopped. Depending is tracing covering only Control Plane or also User Plane messages, estimated 70-500 3 sector LTE BTS can be traced at the same time with one PC. 51

internal use • UEFortrace ( e.g. by using Nemo tool) ©2013 Nokia Solutions and Networks. All rights reserved. LTE RA RD eNB I&V • UDP log file by NWS BTSLog tool • Remote TTI trace for Integrity/Error rates problems

Fault sources •

-

• • • •

-

SW • •

eNB SW bugs

SW not working according to specifications Specifications are ambiguous or lead to faulty behaviour

SCF parameters Suboptimal parameters, missing neighbour relations, etc. UE SW bugs • UE is behaving against specifications or against NSN expected behaviour EPC SW bugs • MME/S-GW is behaving against specifications or against NSN expected behaviour Backhaul SW bugs • Backhaul is dropping packets or sending with too high latency/bursts HW •

Faulty eNB units Faulty/instable UEs

Commissioning

Wrong antenna tilting, loose cables, wrong commissioning parameters, wrong/missing synch sources

Network planning

Too large cells, gaps in geographical coverage, high interference sources, blocking buildings, high mobility Tracing Areas, etc.

Diagnostics

Too heavy tracing or logging causes problems For internal use

52


eNB SW bugs •

•

53

Symptoms No eNB L3 reply for visible UE/EPC signalling No eNB L3 messages sent when expected Rlc Ack visible in call drop after RRC Connection Release RLF or Errors visible in good signal conditions Call drops without any RLF/Errors Huge amount of Ghost Calls Sudden drop of multiple calls at the same time Sudden drop in KPI without parameter changes Sudden drop of KPI after new feature activation Sleeping cells HO rejections RRC connections are reject KPI drop correlation with snapshot fetching, SiteManager connectivity or other normal maintenance actions DSP crashes First step troubleshooting actions Analyze Emil/UE Traces Set triggers and analyze from snapshot TTI Trace and SysLog for unusual behaviour and signal conditions For internal use ©2013 Nokia Solutions and Networks. All rights reserved.

SCF parameters

•

• 54

Symptoms - No UE reply for RRC signalling for multiple UE types - RACH preambles are missed in HOs - Poor signal quality and RLF failures or Errors indicated in certain call phases - Errors distributed evenly in the network - PDCCH blocking visible in counters - Timers expire - High interference visible - Sudden drop of KPI after parameter changes - Call drops correlate to active UE amount - HARQ/RLC/Preamble Retransmissions reach maximum value - Measurement reports don’t initiate HO preparation - HO are started in too poor source cell signal conditions First step troubleshooting actions - Analyze KPI, PM counter and SCF parameter content and For internal use interrelations ©2013 Nokia Solutions and Networks. All rights reserved.

Call drops UE SW bugs

•

•

Symptoms - No UE reply for RRC signalling for one specific UE type - A lot of RLF or Error visible for one specific UE type - KPI drops after new UE type is launched in the network - KPI drops after new UE SW version is launched in the network - UE does not reply in good signal conditions and low loaded cells - No recovery method for dongle disconnect (No Detach messages visible) First step troubleshooting actions - Collect UE logs if possible - Check are the UE types tested in NSN FiVe, IOT or Innovation labs For internal use

55


EPC SW bugs

•

•

56

Symptoms - No reply to S1AP messages - Sudden S1AP Failure indications to multiple eNB - S1 Error indications/Alarms to multiple eNB - Missing S1 HO command forwarding to target eNB - Delayed reply/forwarding to S1AP messages - UEContextReleaseCommand without strange cause values - Duplicate/Strange MME UE S1AP ID allocations - KPI drops after new EPC SW upgrade - KPI drops after EPC parameter upgrade First step troubleshooting actions - FTM mirror port Wireshark trace - FTM logs - EPC logs if possible - In case NSN EPC, contact NSN R&D - InForcase other vendor EPC, contact customer internal use ©2013 Nokia Solutions and Networks. All rights reserved.

Backhaul SW bugs

•

•

57

Symptoms - KPI drops after new Backhaul element type/ SW version/feature is launched or activated in the network - KPI drops after Backhaul parameters/routing version is launched in the network - Link breaks for a cluster of eNB (e.g. behind a specific router) - S1 Error indications/Alarms for a cluster of eNB - Increased latencies in TCP level - Increased TCP re-transmissions - Congestion alarms/indications - Connectivity problems to certain IP addresses/TCP ports - Throughput can not exceed certain (low) limit First step troubleshooting actions - FTM mirror port Wireshark trace - FTM logs - Backhaul elements logs or Wireshark traces, if possible For internal use ©2013 Nokia Solutions and Networks. All rights reserved.

Faulty eNB units

•

•

58

Symptoms - Raw or Customer Alarms indicating faulty/missing units - Known problematic HW versions are installed in the field - KPI drops after new HW installations - KPI drops after storms, floods, earth quakes, etc. natural disasters - Call drops located under certain cell away from cell edge - Long periods/Frequent cell unavailability - Frequent site resets First step troubleshooting actions - Check eNB status with SiteManager - Arrange HW swap - Check that there is no systematic SW/Comissioning/HW version fault For internal use ©2013 Nokia Solutions and Networks. All rights reserved. burning HW components

Faulty UE

•

•

59

Symptoms - On UE type or user in same location is causing frequent call drops - Call drops follow similar pattern in same cell location and during time of day - High interference peaks in same cell location and during time of day - Bursts of call drops in one cell for short time period, without the cell being sleeping/disabled First step troubleshooting actions - Try to find out problematic UE type - Try to find out problematic subscription IMSI - In case thousands of call drops, contact customer and ask them to For internal use ©2013owner Nokia Solutions and Networks. All rights reserved. contact owner of problematic UE

Commissioning

•

• 60

Symptoms - HO fail in target cell for specific eNBs -> frequency sync faulty or missing - Line code violation (and high amount of ghost calls) -> problems in RP3-01 optical fibres - Antenna line alarms - > Loose feeder cables - No S1 connection -> Wrong IP settings - Start-up problems or alarms-> Invalid commissioning parameters or unsupported configuration - Call drops inside certain cell area out side cell edge -> antenna tilting - High DL interference -> Wrong cell power settings or antenna tilting First step troubleshooting actions - Check status with Site Manager - InForcase problems with cabling arrange site visit internal use ©2013 Nokia Solutions and Networks. All rights reserved.

Call drops Network planning •

• 61

Symptoms There are clear holes in network coverage, when analyzed with NetAct Optimizer (or other application, which displays cell locations) There are clear non-line of sight locations for dense populated places Cell under Train, boat or bus traffic are configured with basic HO parameters eNB indicates lack of resources Scheduled UEs per TTI is near maximum continuously during busy hours Timing Advance values indicate too far distances for used carrier frequency and fading environment (e.g. urban city with tall buildings) Call drops in cell edge and no meas report sent Signal levels are low in large cell areas Frequent call drops in same location with multiple UE types High DL interference is certain cell locations First step troubleshooting actions Check status with NetAct Optimizer or site plan For internal use Contact customer for poor coverage ©2013 Nokia Solutions and Networks. All rights reserved.

Call drops Diagnostics

•

•

62

Symptoms - Contention resolution timer expires and some kind of tracing is active - CPU loads are high (> 80 %) despite of moderate Traffic mix - Heavy amount of prints in SysLog or short Snapshot SysLog duration - Emil trace activated during heavy CP traffic - Strange R&D flags visible in swconfig.txt or in Start-up syslog prints - UP data traced/uploaded loaded continuously - Backhaul Traffic >> Air Interface traffic First step troubleshooting actions - Turn of tracing and additional logging - Switch of R&D flags For internal use ©2013 Nokia Solutions and Networks. All rights reserved. - Contact tool developers and R&D

TOOLS

For internal use

63


Data Collected from eNodeB for Analysis BTS Technical Report/BTS Snapshot - The eNB snapshot contains the following information: • Active alarms and alarm history for this eNB • Installed SW versions • eNB configuration (commissioning parameters) • Internal log files for different eNB components (including MAC TTI traces (10 MB)) • PM counter data for this eNB (starting from the last eNB reset, but max for 24 hours) - Logs are frozen by eNB when fetch command is received - Saved in .zip file when loaded - Collection of technical log is triggered by via the • BTS Site Manager protocol. When technical log is collected from the Site Manager, the term snapshot is used as a synonym. For internal use 64


Measurement Tools (NSN R&D Internal tools) The following measurement tools have been developed by NSN for R&D that you can also use • BTSLog • BTSlog is a testing tool including functionality of: • • • •

System log recording & real-time monitoring tool Message trace and recording R&D parameters handling BTSlog is used for BTS testing and works with Windows

• LTE Browser • LTE Browser is used to get the real time data about the CPU load, & real time UL PHY measurement data (SNR, BLER etc) memory dump information's from DSPs and static Cell specific configuration information

• Emil • • • •

EMIL is an offline message tracing analyzer tool. EMIL parses all the internal messages between different software components, EMIL has the capability to show all ASN.1 messages (RRC, S1 AP, X2 AP messages) Emil L3 decoding works even if IPsec is used for S1/X2

• TTI Trace Analyzer 65

• Traces Cell and User specific TTI information For internal use • Helps to analyze the behavior of the DL/UL Packet . Networks. All rights reserved. ©2013 Scheduler Nokia Solutions and • Is implemented as ring buffer which is filled up with information all the time

For internal use

66


Analysing BTS message logs with Emil

After Sack and profile configuration trace file can be opened with EMIL.

For internal use

67


LTE BROWSER • Definition: noise rise NR = (I+N)/N, I = interference, N = thermal noise • SNR0=S/N (only noise, no interference) • SNR1 = S/(I+N), (noise + interference using eg external rf generator) • NR = SNR0/SNR1 = (I+N)/N, assuming average signal power S is the same • Example: SNR0=13dB, SNR1=8dB NR= 5dB

Real-time online measurement from eNB LTE Browser

SNR0 Noise rise SNR1 Interference source is off (only thermal noise) For internal use 68

Interference source is on ©2013 Nokia Solutions and Networks. All rights reserved.

LTE Test Tools / 2010 May / Jaakko Kärnä

TTI Trace Analyzer

For internal use

69

©2013 Nokia Solutions and Networks. All rights reserved. /

ToC

UL HARQ Support in TTI Trace [3GPP-36.312 chapter 8] specifies the relationship of the timing between eNB and UE with the following wording: “For FDD and normal HARQ operation, the UE shall upon detection of a PDCCH with DCI format 0 and/or a PHICH transmission in subframe n intended for the UE, adjust the corresponding PUSCH transmission in subframe n+4 according to the PDCCH and PHICH information”.

8 HARQ processes per UE

Predefined RV sequence for successive packet transmissions

Non-Adaptive HARQ retransmissions use the same MCS, the same rate matching and the same PRBs

Unlike DL , HARQ process id not sent in UL because the UE is expected to retransmit For in exactly 8 TTI. internal useACK/NACK is sent by eNB in exactly 4 TTIs.

70

Retransmission is sent for Nack or DTX exactly after 8TTIs. This is synchronous HARQ procedure. ©2013 Nokia Solutions and Networks. All rights reserved.

RRM in TTI traces & related System Performance - Shomik Pathak - Nov 2011

WIRESHARK

For internal use

71


Traffica for eNodeB, what and how? • Traffica for eNodeB provides • Real-time monitoring solution for subscribers radio interface experience

• Direct interface to NSN Network element • Subscribers radio interface experience • Includes detailed internal and external error causes for root cause analysis

How Traffica for eNodeB works • Traffica receives data reports based on actions Users Equipment in the network. •

User Equipment specific report is sent in real time immediately to Traffica when the triggering event happens.

• RRC/eRAB activities by User Equipment will trigger a data report to sent to Traffica. • 72

RRC or eRAB establishment/release and related failures.

For internal use


• Handover and Handover attempts will trigger a data report to be sent to Traffica.

Traffica for eNodeB, use cases • Real-time view to Radio events

• Drive Test Minimizing

– Detect Radio failures as they occur – Visibility to events, locations, failure

– Cell-ID and UE distance from eNodeB included in the Traffica report, UE location can be specified quite well.

causes

– Locations with most events or failures

– At handover, UE measurements from each neighbour cell (field strength) is available

– Monitor Number of call setups / cell

• Visibility Handovers

– Traffica can count normal and

– Monitor Handover events – Visibility to Handover failures and

abnormal call setups and releases according to each location

causes

– Handover locations and field strength

• Location based analysis – events and failures per location

Fast root cause analysis via detailed internal error cause descriptions

For internal use

73


Data Collected from eNodeB for Analysis LTE1053 Traffica for LTE - Traffica receives event based reports from Trace Data Collector in real time. - Traffica analyses the data and creates KPIs based on the data and visualize in Traffica real-time graphs. General (graph set): • • • •

•

Handover Analysis (graph set) • • •

•

74

Incoming HO types (Incoming HO per cell, Incoming HO types, UE distance measurements, source cell RSRP, Target cell RSRP) Outgoing HO types (outgoing HO per cell, outgoing HO types, UE distance measurements, source cell RSRP, Target cell RSRP) Handover failures (outgoing HO failures / cell, HO failure causes, HO failure phases/cell)

Troubleshooting (graph set) • • • •

•

RRC overview (setup failure, active failure, setup attempts, normal releases, setup failure distribution, etc) RRC service failure Details(RRC setup events, RRC setup events / establishment cause, RRC setup failure distribution /cause) RAB overview (setup failure, active failure, setup attempts, normal releases, setup failure distribution, etc) RAB service failure Details(RAB setup events, RAB setup events / establishment cause, RAB setup failure distribution /cause)

RRC setup failures(RRC setup failures per cell, RRC setup failure sources /cell) RAB setup failures (RRC setup failures per cell, RRC setup failure sources /cell, UE distance at RAB setup failure) RAB active failure(RAB active failure per cell, RAB active failure sources per cell, UE distance at active failure) Handover failures (outgoing HO failures / cell, HO failure causes, HO failure phases/cell)

Triggers for receiving these reports are • • •

RRC or RAB establishment / release and failures related to those Intra-system oruse Intersystem handover, successful or failure includes failure cause information. For internal The interface between the Trace Data Collector and Traffica is event reportsand based (TCP/IP). ©2013 Nokia Solutions Networks. All rights reserved.

NetAct Tracing for LTE

Introduction - NetAct Trace Viewer provides the tracing functionality for LTE - The same application can be used for LTE tracing as for 2G and 3G tracing. - LTE tracing allows to monitor calls of subscribers in LTE network - Functionality aligned with Call Trace Analyzer

NetAct

eNB

Trace Viewer

MME eNB For internal use

75


NetAct Tracing for LTE

Use Cases - Network Rollout and new installation • Monitor calls of field engineers on-line • Guide drive testing team at the problem spot on a field instead of testing just pre- planed route • Get fast reply after software and hardware upgrade about network coverage by monitoring on-line field engineers • Act promptly to corporate customer network coverage related problems • Find out right problem related to case • Reduce time used to network coverage related cases

- Detailed investigation of subscriber complaints - Solving problem cases related to network coverage - Features and trace content for • • • •

Site & Cluster acceptance Change acceptance Saving test results In depth analysis For internal use

76


Measurement Tools – Terminals (Drive Testing) • The following drive testing tools using a terminal are recommended: • Agilent/JDSU E6474A (Currently most cost efficient solution and supports samsung) • Currently support LG Data card and partial Samsung support • Support soon also Qualcom Chip set • Supports up to eight frequency bands • NEMO Outdoor from Anite (No samsung support) • LG LTE L-1000 and L-2000 chipset support • LG G7, G13 and G17 UE support • PCTEL EX LTE Scanner and R&S TSMW LTE scanner supports • Support soon also Nokia Modem RD-3 and QUALCOMM LTE chipsets (MDM9200 / MDM 9600)

• Rhode and Schwarz (ROMES) R&S®ROMES4 is a test platform for mobile measurements in all modern radio networks. In combination with scanners and test mobile phones, it forms a complete system for coverage and quality-of-service (QoS) measurements.

• Accuver XCAL • XCAL is the drive testing solution from Accuver • Supports LG and Samsung UE For internal use 77


https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/WebHome LINK FOR LTE TOOLS

For internal use

78


SON FOR LTE Self-Configuration

• Flexibility in logistics (eNB not site specific) • Reduced site / parameter planning • Simplified installation - less prone to errors • No / min. drive-tests • Faster roll-out

Self-Optimization

• Increased network quality and perform. • Parameter optimization • Reduced maintenance, site visits

Self-Healing

• Error Self-detection and mitigation • Speed up maintenance • Reduce outage time

For internal use

79


Self-Configuration for LTE

Centralized SON

• SON Plug and Play • Flexi BTS auto-configuration • Automated Physical Cell ID assignment

• Automated Neighbor Relations (ANR) Management • Automated neighbor list configurations of LTE, inter-frequency LTE, inter-system WCDMA and GSM neighbor cells • Automated update of neighbor list configuration in case of GSM and WCDMA neighbor cells removals / additions

Distributed SON

• PRACH Management SON Plug and Play •• Automatic of PRACH parameters by Optimizer Flexi BTS assignment auto-connection • Flexi BTS auto-configuration • LTE Certification Authority and Registration Authority Management

For internal use

80

• Automated Neighbor Relations (ANR) – Fully UE based • LTE, inter-system neighbor list configuration due to UE ©2013 Nokia Solutions and Networks. All rights reserved. found/measured neighbor relations

Self-Optimization for LTE

Centralized SON

• Automated Neighbor Relations Management • Optimization of found neighbor relations by blacklisting inefficient ones

• Energy Saving • Centralized management for Flexi BTS shutdown and energy saving features

• Mobility Robustness Optimization • Optimizing Intra-frequency radio network HO-configuration in order to reduce too early, too late, ping pong HOs or HO to wrong cells

Distributed SON

• Service Quality Optimization •• Minimization of Drive Management Tests Automated Neighbor

•• Geographical analysis ofof location specific performance based on call Support for optimization neighbor relations level statistics

• Energy Saving

• Intelligent Flexi BTS shutdown • Enhanced power saving • BTS power saving mode

• Load Balancing For internal use

81

• Physical HO optimization • Real-time load balancing ©2013 Nokia Solutions and Networks. All rights reserved. • Minimization of Drive Tests

• Support for call level statistics

Self-Healing for LTE

Centralized SON

• Cell Outage Detection • Cell outage detection by KPI and alarm supervision

• Cell Outage Triggered Reset • Cell outage detection triggered reset of cells

• Cell Outage Compensation • Coverage hole compensation by antenna adjustments

Distributed SON

• Automatic Interface Alarm Correlation • Alarm correlation in case problems Reset of a central interface (X2, • Cell Outage Detection andofTriggered S1) and or entity • KPI alarm support

• Cell Outage Reset • Automated cell lock & unlock with escalation to reset

• Cell Outage Compensation • Emergency configurations to compensate neighbor cell outage For internal use

82


BACK RL40

Optimizer for LTE Effortless self-configuration and optimization on Network Layer to maximize the network performance Optimizer in the core of the Self Organizing Network solution in NetAct for centralized SON Self-Optimisation: ▪ Auto-tuning of neighbour cell lists and other key parameters based on UE and eNB measurements on network management level ▪ Cell outage compensation ▪ Able to cope with a single vendor domain as well as with multi-vendor access networks

Optimizer NetAct Network Management Optimizer Domain Manager

Nokia Siemens Networks

For internal use

83


CM PM

Vendor X

CM PM

Physical-layer Cell ID (PCI) Management - PCI management done in similar way than UTRAN PSC - PCIs are not unique - initial allocation of PCI values • confusion-free: all neighboring cells need to have different PCI values • collision-free: no two cells must be received with same PCI - minimum reuse distance algorithm - optional: neighbor’s neighbor have different PCIs • Support for synchronized LTE networks - three additional rules to allocate PCIs • cells under one eNB shall have PCIs from the same PCI group(s) • neighbor cells shall avoid same PCI modulo 30 • neighbor cells shall use odd and even PCI groups evenly (approximation of a modulo 6 rule) PCI group, N(1)ID 0 1 2 … 165 166 167

For internal use

84

©2013 Nokia Solutions and Networks. All rights reserved. Achim

0 0 3 6 … 495 498 501

PCI offset, N(2)ID 1 1 4 7 … 496 499 502

2 2 5 8 … 497 500 503

PCI vs. eUTRAN Cell Global ID (ECGI) eUTRAN Cell Global ID • • • •

Part of System Information Block 1 in BCCH Unique Cell Identity in the with PLMN and eNB Identities Long repetition cycle of 80 ms UE does not decode this if not advised to do !!!

Physical-layer Cell ID

For internal use

• Part of the Synchronization Sequence/Reference Symbol • Limited to 504 values, not unique • Short repetition cycle of 1 ms • fast, no DRX needed • careful assignment needed because a UE shall never receive the same value from 2 different cells -> no collisions, no confusions

85

©2013 Nokia Solutions and Networks. All rights reserved. Achim

Physical-layer Cell ID (PCI) Management - Need to have clear identity codes to ensure reliable neighbour cell dependencies - Similar structure/management to UTRAN Primary Scrambling Codes

14 23

412 66

1

- PCI allocation algorithms: • confusion/collision free allocation • minimum reuse distance in allocation criteria • user can control the algorithm - option to allow neighbor’s neighbor have different PCIs - minimum reuse distance can be defined

• support for synchronized LTE networks - cells under a eNB shall have PCIs from the same PCI group(s) - 504 PCIs have been grouped into 168 groups, 3 codes in each - neighbor cells shall use odd and even PCI groups evenly 86

For internal use - Optimizer 3.0 assigns initial PCI values©2013 in Nokia autoSolutions and Networks. All rights reserved. configuration process

Presentation / Author / Date

511 234 322

98

LTE PCI Management Enhancements - If LTE eNodeBs are deployed and managed with various processes it can happen that PCI collisions or confusions occur • When LTE Auto-configuration process is used for eNodeB roll-out PCI conflicts never happen

• PCI conflict/confusion analysis can be activated in Optimizer – Collision analysis for selected set of cells • Collisions and confusions checked against the user defined criteria – – Analysis against violations of additional constraints is run too

14 23 1

free allocation for PCIs by minimising the amount of changes Cells with most serious PCI problems are changed first For internal use ©2013 Nokia Solutions and Networks. All rights reserved.

234 322

98

– Analysis results are presented in table views and on geographical map • PCI reallocation can be activated in Optimizer – Resolution can be activated for the selected area – Optimizer finds the collision/confusion and violation

Presentation / Author / Date

412 66

Forbidden PCIs (in Optimizer 3.0 CD2)

87

511

Automated Neighbor Relations (ANR) in LTE

• The handset mobility relies on information given by neighbor cells and the quality of it • The ANR function automates and optimizes that handover process with real-time information exchange between the parent cell (source) and receiving cell (target) • The automated neighbor relations remove manual planning, maintenance and subsequently errors from the process The steps of ANR process • Neighbor cell discovery • Neighbor Site’s X2 transport configuration discovery • X2 Connection Set-up with neighbor cell configuration update For internal use 88


Key Benefits of ANR • The automated detection and configuration of unknown cells and respective sites supports the self configuration of the neighbor cell information without operator involvement Automation of adjacency planning & maintenance leads to O&M cost reductions The error free ANR process simplifies and speeds up the network rollouts and expansions for faster time to market Expected amount of LTE neighbor sites is 6-32

For internal use

89


ANR support for LTE with OAM extension (LTE492) ANR support for LTE with OAM extension (LTE492) new mechanism allowing for identification of unknown UE-reported neighbour cells each newly activated Flexi Multiradio BTS is provided with table that maps Physical Cell Id & DL EARFCN of adjacent cells to C-plane IP address of their parent eNBs PCI/RF/IP@ mapping tables are generated by NetAct based on the neighbour site relations analysis and downloaded to BTS during autoconfiguration procedure based on PCI/RF/IP@ mapping table, eNB is able to determine adjacent eNB hosting unknown cells and to retrieve from it necessary (in the context of neighbour relations) configuration information

90

neighbour site and neighbour cell configuration information is derived via X2 For internal use ©2013 Nokia Solutions and Networks. All rights reserved. connection

ANR support for LTE with OAM extension (LTE492) ANR support for LTE with OAM extension (LTE492) X2 connection setup procedure is triggered by UE measurement reports containing PCI that is not known to eNB as a neighbour cell but is listed in PCI/RF/IP@ mapping table X2 connection establishment is not based on info from MOC ADIPNO only as it was with LTE724

target eNB accepts incoming X2 connection setup requests from a source eNB even if no IP-address of the source eNB is configured on the target site no need for configuration of IP@s at both ends of the X2 connection as it was with LTE724

Feature benefits

91

automated detection and derivation of configuration information of unknown neighbour sites and cells no need for manual pre-planning of eNB adjacencies for X2 connection setup self-configuration of the neighbour cell information without operator For internal use involvement - reduced planning efforts ©2013 Nokia Solutions and Networks. All rights reserved.

LTE 771 Optimization of neighbor relations

Evaluating of handover behaviour – blacklisting proposed ? • NetAct Optimizer supervises all registered Optimizer cell relations between neighboring LTE cells if they are still valid and reliable candidates to Configurator be a handover destination • When the outcome results in an inefficient neighbor relation, the according cell relation is blacklisted for handover • Blacklisting can be done manually by the operator as well • White listing (excluding optimization) can be No HO permanently done manually by the operator • The changes, proposed by optimization algorithms, can be taken over automatically internal use (without For any operator intervention) or after ©2013 Nokia Solutions and Networks. All rights reserved. 92 confirmation by the operator

CM PM

RL30

ANR InterRAT UTRAN / GERAN

Automated configuration: NO manual work for neighbor cell relation information Optimizer

NetAct

• Central neighbour relation establishment with NetAct Optimizer • Based on geo-location: no ANR InterRat supporting UEs needed

CM Configurator N-itf LTE GERAN UTRAN

UTRAN/GERAN Domain Managers

UTRAN

GERAN

For internal use

93

Feature ID(s): LTE 783 / 784


CM

Technical Details •

Table of Contents

LTE783 ANR Inter-RAT UTRAN / LTE784 ANR InterRAT GERAN LTE site/cell (source of NBR) and RAT site/cell (target of NBR) must be under management of NetAct For both (source and target), planning information must exist in NetAct Configurator database, i.e.: geo-location and antenna direction data NetAct Optimizer uses this planning data to calculate: Distance between LTE cell and RAT cell

NetAct

Relative main lobe directions Is determined based on antennas directions of source and target cell

Based on them (distance part and relative main lobe directions part), priority for each adjacency is calculated. Priority gives information about usefulness of each NBR.

CM CM

LTE Domain Managers

CM

UTRAN/GERAN Domain Managers

LTE source

Also following inputs must be defined: Search distance between LTE cell and RAT candidates for excluding far away cells Minimum/maximum number of NBRs from each considered LTE cell For internal use Otherwise default values are considered ©2013 Nokia Solutions and Networks. All rights reserved.

This information must be available as precondition before starting of NBR planning by LTE783/784

Optimize r

Configurator

Is estimated based on geo-location of LTE cell’s antenna (source) and geolocation of RAT cell’s antenna (target)

94

Main Menu

ANR Inter-RAT in details – 1/8

UTRAN target GERAN target

CM

Main Menu

For internal use

95

©2013 Nokia Solutions and Networks. All rights reserved. MBB CS Network Engineering / Bartlomiej Bednarz

Mobility Robustness Optimization Input/Output Input Parameters

PM counters • HO Attempt/Success Rate • RLF rate Dedicated new counters (based on 3GPP Rel9 Radio Link failure indication and handover report messages on X2-interface) • Too early HO of type 1 counts attempted failed handovers with subsequent reestablishment • Too early happening HO of type 2 counts several kinds of problems in handover target cell quickly after a completed handover • Too late HO counts missed handover attempts ending up in radio link failure Output parameters:

96

Modified values for For internal use Offset (CIO) (per NR pair) • Cell Individual ©2013 Nokia Solutions and Networks. All rights reserved. • Time-to-trigger (A3 time to trigger) (per cell for all NR)

Recommendation to use SON

1. Consider SON expert from the beginning of roll-out procedure plan to check the advantage of SON. 2. Set-up the correct communication line between global and local. Involvement of SON expert and product line is strongly recommended to select the SON solution proper and optimized for customer network environment. 3. Need the engineers who get end-to-end training about SON(not only eNB SON and NetAct optimizer) because SON issues should be investigated end to end.

For internal use

97


Thank you

98

For internal use /


Lte Operations and Maintenance

Recommend Documents