DYNES-Deployments

["Gray, Charles" <>]

Are there any further developments to report regarding DYNES and circuit creation since my post on 8/21/13?  I haven’t seen any further posts to dynes-deployments nor any updates to the DYNES website on current status.

 

Today when logged into the FDT server as admin and issuing a “dynesfdt ping uf”, the session throws a java error as shown below.  It appears to fail at creating a circuit that was created successfully (with no java errors) when tried a couple of weeks ago.

 

Issuing a “dynesfdt list” also throws a java error (permission denied) before providing a list of peer sites.  Is the java error something that Tufts needs to fix?

 

TIA,

Charles Gray

Senior Systems Administrator

Tufts Technology Services (TTS)

169 Holland Street

Somerville, MA 02144

(617) 627-5327 (w)

(617) 627-3307 (f)

it.tufts.edu

 

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:18:40 PM fdt.agent.session.CreateCircuitTask waitForIDCCreate

INFO: Waiting for reservation tufts.edu-2137 -- Current OSCARS/IDC status: INPATHCALCULATION; Deadline in: 18 minutes 0 seconds

 

Sep 05, 2013 2:18:49 PM fdt.agent.config.FDTAgentConfigCheckTask run

WARNING: [ HANDLED ] FDTAgentConfigCheck not updated cfgPointer. Cause:

java.io.FileNotFoundException: /opt/FDTAgent/conf/fdtagent-running.conf (Permission denied)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at java.io.FileWriter.<init>(FileWriter.java:90)

        at fdt.agent.common.Utils.storeOptionsToFile(Utils.java:872)

        at fdt.agent.config.FDTAgentConfigCheckTask.run(FDTAgentConfigCheckTask.java:127)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:18:50 PM fdt.agent.session.CreateCircuitTask waitForIDCCreate

INFO: Waiting for reservation tufts.edu-2137 -- Current OSCARS/IDC status: INSETUP; Deadline in: 17 minutes 50 seconds

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:01 PM fdt.agent.session.CreateCircuitTask waitForIDCCreate

INFO: Waiting for reservation tufts.edu-2137 -- Current OSCARS/IDC status: ACTIVE; Deadline in: 17 minutes 40 seconds

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:01 PM fdt.agent.session.CreateCircuitTask waitForIDCCreate

INFO: Checking VLAN info into the path for tufts.edu-2137 SUCCEEDED! -- Current IDC Status: ACTIVE

Sep 05, 2013 2:19:01 PM fdt.agent.session.CreateCircuitTask waitForIDCCreate

INFO: Found VLANs (SRC="2085," DST=733) for tufts.edu-2137 in 1 interations

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.CreateCircuitTask run

WARNING: OSCARS/IDC status 'OK' to be notified to agent session

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession circuitCreationFinished

INFO: [AgentSesion] circuit creation finished OK. Details:

IDCCircuitDetails [resDetails=net.es.oscars.wsdlTypes.ResDetails@557d5f94, createReply=null, operStatus=OK, statusDescription=Status from IDC: ACTIVE, errorCode=0]

Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession circuitCreationFinished

INFO: Will get the VLAN information from PathInfo

Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession circuitCreationFinished

INFO: [ AgentSession ] [ circuitCreationFinished ]  src vlan: 2085; remote vlan: 733

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession commitLocalResources

INFO:

 { Virtual interface } Executed:

/usr/bin/sudo /sbin/vconfig add eth0 2085 Resp : Added VLAN with VID == 2085 to IF -:eth0:-

OK

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession commitLocalResources

INFO:

 [ commitLocalResources ] { Virtual interface IP } Executed:

/usr/bin/sudo /sbin/ip address add 10.53.128.5/32 brd + dev eth0.2085 Resp : OK

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession commitLocalResources

INFO:

 [ commitLocalResources ] { Virtual interface UP } Executed:

/usr/bin/sudo /sbin/ifconfig eth0.2085 up Resp : OK

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession commitLocalResources

INFO:

 [ commitLocalResources ] { Virtual interface ROUTE }:

/usr/bin/sudo /sbin/ip route add 10.53.224.5/32 via 10.53.128.5 dev eth0.2085 Resp : OK

 

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:06 PM fdt.agent.session.AgentSession handleCommitIPResp

INFO: [handleCommitIPResp] IDC Circuit is up; local and remote resources commited. Trying to test remote connectivity with ping

 

Sep 05, 2013 2:19:09 PM fdt.agent.config.FDTAgentConfigCheckTask run

WARNING: [ HANDLED ] FDTAgentConfigCheck not updated cfgPointer. Cause:

java.io.FileNotFoundException: /opt/FDTAgent/conf/fdtagent-running.conf (Permission denied)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:212)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:165)

        at java.io.FileWriter.<init>(FileWriter.java:90)

        at fdt.agent.common.Utils.storeOptionsToFile(Utils.java:872)

        at fdt.agent.config.FDTAgentConfigCheckTask.run(FDTAgentConfigCheckTask.java:127)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:09 PM ExtProccess stdout

INFO: PING 10.53.224.5 (10.53.224.5) 56(84) bytes of data.

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:09 PM ExtProccess stdout

INFO: From 10.53.128.5 icmp_seq=2 Destination Host Unreachable

 

Sep 05, 2013 2:19:09 PM ExtProccess stdout

INFO: From 10.53.128.5 icmp_seq=3 Destination Host Unreachable

 

 

[ FDTAGENT ] [ INFO ] Sep 05, 2013 2:19:09 PM ExtProccess stdout

INFO: From 10.53.128.5 icmp_seq=4 Destination Host Unreachable

 

From: Brian Cashman []
Sent: Thursday, August 22, 2013 2:46 PM
To: Gray, Charles
Subject: Re: FW: [dynes-deployments] Need some OSCARS/DYNES/<something> help...

 

Charles,

Yes, we're going to update the documentation further once testing reaches a state in which circuits are mostly created successfully and once the last set of sites to receive equipment have been configured. We're not there now but we're making progress.

Thanks,
     Brian

On 8/21/13 1:40 PM, Gray, Charles wrote:

Brian,

 

Thanks for the info.  Dynesfdt ping of UF did indeed work.  What is the status of the other peer sites re: Tufts?  Is this something readily available/updated?  Guess one way to answer the question would be to attempt a ping of each peer site.

 

So, now if I want to “venture further” into the fray is there more information regarding circuit setup beyond the “FDTAgent for DYNES” document.  I’ve noticed there have been no recent updates to https://spaces.internet2.edu/display/dynesdoc/DYNES+User+Documentation of examples.  Are further doc updates expected?

 

TIA,

-Charlie

 

From: Brian Cashman []
Sent: Wednesday, August 21, 2013 1:18 PM
To:
Cc: Gray, Charles
Subject: Re: FW: [dynes-deployments] Need some OSCARS/DYNES/<something> help...

 

Charles,

At this time, we're still testing connectivity between sites. The only test we're running right now is between Tufts and the University of Florida (both directions). So, dynesfdt ping uf should work. The others are still being tested as I mentioned and we'll be adding more sites to the automated tests hopefully soon.

The thread below was specific to Rice and Houston, two sites that happened to have S4810 switches. A while back the DYNES-PIs decided to replace DRAGON in as many places as we could along with upgrading the OSCARS version from 0.5 to 0.6.

Sites with S4810 switches will have OESS software installed
Sites with 8024F switches (e.g., Tufts) will have the Bridge-PSS software installed

OESS and Bridge-PSS replace DRAGON. At Tufts, we've upgraded OSCARS to 0.6 and installed the Bridge-PSS. So, no DRAGON. I did happen to see that the VM running OSCARS and the Bridge-PSS is still named "tufts_vlsr_idcv6," which I think was the name it had originally when it was running DRAGON.

Glad to see you back into the fray.

     Brian

On 8/21/13 12:49 PM, Gray, Charles wrote:

I’ve been out of the DYNES fray this summer and focused on a physical reorg of our datacenter.  Having just started looking at DYNES again, I reread the deployment list emails and noticed this thread wondering if these circumstances were applicable to Tufts.  If so, I’d appreciate advice on how to articulate what I need to the list.

 

I’ve tried to DYNESFDT ping various sites with no success.  Could still be pilot error needing more education, but I noticed that DRAGON was also installed on Tufts’ IDC.

 

Regards,

Charles Gray

Senior Systems Administrator

Tufts Technology Services (TTS)

169 Holland Street

Somerville, MA 02144

(617) 627-5327 (w)

(617) 627-3307 (f)

it.tufts.edu

 

 

 

From: [] On Behalf Of Brian Cashman
Sent: Wednesday, July 03, 2013 11:50 AM
To:
Cc: William Deigaard; Jason Bothe; Deniz Gurkan; Charles E Chambers; ; Ty Bell
Subject: Re: [dynes-deployments] Need some OSCARS/DYNES/<something> help...

 

William and Deniz:

I didn't realize until today that both of your sites, Rice and Houston, have an S4810 switch. Rather than spend too much time on this particular issue let me suggest a hopefully better alternative. We've been having problems with DRAGON setting up circuits and DRAGON, at least the DRAGON we installed on your IDCs, supports that switch. What I'd like to do is instead install OESS software and if it's not on there already, a version of the switch firmware that supports OpenFlow. The OESS software will include an updated version of OSCARS and replaces DRAGON with configuring the switch using OpenFlow. If that's acceptable to you please let me know. Ty Bell, cc'd here, would be the one doing the installation.

The IDC I have for Rice is 168.7.170.198. Does Houston have a separate IDC?

Thanks,
     Brian

On 7/2/13 12:43 PM, William Deigaard wrote:

Hello,

 

At Rice (and UofH), it appears, we cannot seem to create a circuit reservation on DYNES

 

Perhaps we broke something.  Perhaps we're doing something wrong.  Perhaps something never worked.  We can't quite tell.

 

When we try to create a circuit using the OSCARS interface, we get errors like the following:

 

 

Errors on Oscars console for circuit from Rice to LEARN:

 

 

 

 

Errors in Oscars ConnectivityPCE.log

 

level=INFO ts=2013-07-02T04:02:16.598000Z event=net.es.oscars.pce.Connectivity.pceCreate.start guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 
level=INFO ts=2013-07-02T04:02:16.598000Z event=net.es.oscars.pce.Connectivity.pceCreate.end guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 status=0 
level=INFO ts=2013-07-02T04:02:16.869000Z event=net.es.oscars.pce.Connectivity.invoke-client.start guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 msg=getTopology url="https://localhost:9019/topoBridge 
level=WARN ts=2013-07-02T04:02:18.437000Z event=net.es.oscars.pce.Connectivity.invoke-client.end guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 errSeverity=MINOR status=-1 msg="OSCARSSoapService.invoke:Exception connecting to getTopology on https://localhost:9019/topoBridge Message is: A SOAP 1.2 message is not valid when sent to a SOAP 1.1 only endpoint." 
level=ERROR ts=2013-07-02T04:02:18.438000Z event=net.es.oscars.pce.Connectivity.calculatePath.end guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 errSeverity=MAJOR status=-1 msg="Error from topoBridge: OSCARSSoapService.invoke:Exception connecting to getTopology on https://localhost:9019/topoBridge Message is: A SOAP 1.2 message is not valid when sent to a SOAP 1.1 only endpoint." 
level=INFO ts=2013-07-02T04:02:18.438000Z event=net.es.oscars.pce.Connectivity.invoke-client.start guid=dynes.rice.edu-WBUI-a92dba30-6c55-41b3-9744-5c4be17f7da6 gri=dynes.rice.edu-307 msg=PCEReply url="https://localhost:10000/OSCARS/PCERuntimeService 

In particular, we've noticed that we can't seem to make a basic connection to the topoBridge:

[root@vm1-idc-rice oscars]# wget https://localhost:9019/topoBridge

--2013-07-02 16:34:47--  https://localhost:9019/topoBridge
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:9019... connected.
OpenSSL: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
Unable to establish SSL connection.

 

 

Any suggestions from folks out there?

 

Thanks.

 

 

-- 
William Deigaard  ^-^
() (O O)
Go Owls!         ( v )
------------------m-m--Director,
713-348-5262 (o) Networking, Telecommunications & Data Centers
713-487-7423 (m) Rice University