Server Busy Error in SIEBEL

"Server you are trying to access is either busy or experiencing difficulties” 
Undoubtedly, for any Siebel admin this is the most common issue to troubleshoot. In this post I will try to list down most possible reasons causing this issue.

1)      Issue with Database

Check database connectivity by using database client like sqlplus/plsql developer/Toad

2)      Issue with Data source name i.e. DSN

Siebel connects to database using DSN so check

In UNIX:

       Setup environment variables, go to bin directory & run below command

       odbcsql /u sie_app /p ******** /s siebsrvr_CMS

       Where,

       /u - database user

       /p-  database password

       /s-   Datasource Name

       If this is not working then you have to check .odbc.ini file.
       especially driver name & TNS service name (Same service name must be present in tnsnames.ora file)

If above command errors out then check following environment variables

echo $ORACLE_HOME

echo $LD_LIBRARY_PATH

          echo $TNS_ADMIN

           In Windows:

           Use ODBC datasource administrator -> System DSN -> Double click on Data source ->Test     

           connect

3)      Issue with database account

Make sure you can access database with following accounts

1)      Anon user : In case of DB authentication

2)      SADMIN/LDAP USER: In case of LDAP or SSO Authentication (assuming you use SADMIN/LDAP USER as common login to connect to database )

3)      Check anon user and SADMIN/LDAP user is present in S_USER table.

select  * from S_USER where login like ‘SADMIN’

select  * from S_USER where login like ‘ANONUSR’

4)      Check Siebel services

           Ps –ef | grep sieb

           ps –ef | grep siebsvc

           Check Siebel server & Gateway is running.

           list_ns

           list_server all

5)      Check SC broker & AOM status

set server siebapp

list comp scbroker show CC_ALIAS, CP_DISP_RUN_STATE

list comp SSEObjMgr_enu show CC_ALIAS, CP_DISP_RUN_STATE

list tasks for comp SSEObjMgr_enu show TK_TASKID, TK_DISP_RUNSTATE, TK_LABEL

list tasks for comp scbroker show TK_TASKID, TK_DISP_RUNSTATE, TK_LABEL

6)      Check port 2321 of application server

telnet tsgsq2902 :2321 (from web server)

netstat  -an | grep 2321 (from application server)

7)      Check core services are running

cd  $MWHOME/bin

./mwadm status

Or check status of process regss , watchdog ,mwrpcss etc   

8)      Check data source definition

srvrmgr:siebapp> List param DSConnectString for named subsystem ServerDataSrc show PA_ALIAS, PA_VALUE

9)      Check CPU/Memory/Mount space  utilization

df -hk

top –n 0

prstat –a 5 5

10)    Try using vanilla Siebel.srf file

11)    Try using backu[ copy of siebns.dat file

13)   Check lbconfig.txt file & eapps.cfg to make sure requests are reaching to SCbroker

a)      lbconfig.txt location specified correctly in eapps.cfg under communication management section

b)      virtual host value must be set to true

c)       server connect strings must be given properly in lbconfig.txt file

d)      object manager connect string in eapps.cfg must contain value Virtual host

                  

14)   Check application server log files, WEB instance log, SWSE logs, SSO agent logs

15)   If it is newly installed environment then check environment by using EVT

16)   Make sure that all the mounts including File system are available.

17)   Restart Application Server

delete .osdf  file make sure that  .shm file is getting deleted after stopping server, if not, delete it manually.

18)   Take web server bounce

19)   Take clean bounce to recreate .svc file

Refer article http://www.siebeloracle.com/how-to-take-clean-bounce-of-siebel-server/ for clean bounce

20)   Verify messages in /var/adm/messages

21)   Physical host reboot

This should be the last resort, if nothing else works. Server reboot will open clogging of ports and will restart all the OS processes.