New: LMB not working on Production
- 15:15 Loaded flat files for Summer 2011 and Fall 2011
- 14:20 Posted message to Vancko Hall site news
- 14:10 Posted message to Twitter
- 13:45 Checked LMB log (last entry: 2011-07-07T16:11:28)
- 13:15 Verified that pin changes were not going through to Vancko Hall
- 08:01 E-mail notification "Moodle has not received a message from Luminis Message Broker in 950minutes"
New: LMB now working again on both prod and test. Note from MR: Digicert had indeed changed their certificate chain. They kept the old one, though, and it is still available to us to use -- it is just that they changed the default intermediate certs. We've restored the original cert chain.
- We will be running the flat files again at about 14:45 for the final time, then turn the LMB live feed back on.
Continued: LMB working on Test, not working on Production, running flat files daily after 3 p.m.
The issue is mainly affecting BSN "hidden assignments" in courses - where fellow instructors are added as hidden assignments, and these are turned into regular non-hidden assignments each time we run the flat file - because those assignments are coming in from the Banner export each time. This means the BSN (or any instructor with hidden assignments) may need to visit "Assign Roles" - remove any non-hidden assignments and re-add them as hidden assignments.
- We have continued daily contact with Moodlerooms regarding the issue, and they are pursuing a support request with Digicert regarding changes made to their SSL Chain Certificate. They will keep us apprised as it progresses.
New: LMB working on Test server again
Continued: LMB not working on Production
- a.m. LMB messages being received on test, confirmed PIN change to test
- 10:41 Updated ticket to MR - with SSL error on Prod.
- 08:30 LMB server re-configured and confirmed working
New: LMB not working on Test server
Continued: LMB not working on Production
- 15:20 Flat files saved from LMB and processing on VH and Test servers
- 14:50 Notice posted on VH site news
- 11:06 LMB administration issues on our end - working to restore. We will run flat files for Summer 2011 and Fall 2011 each day until the live web services are back in order.
- Updated MR Ticket: LMB status on Test says last message: (Sunday, 19 June 2011, 12:16 PM), Prod: (Friday, 17 June 2011, 11:08 AM)
- 09:00 tested PIN change, not working on either Prod or Test
Continued: LMB not working on Production.
- 10:14 reply from Level 2 MR - looking further into issue.
- 07:58 reply to Level 1 MR that this is related to live import web services and not XML import, request to up to Level 2/ Silverman
- 07:50 re-confirmed that the Prod LMB status page has not received LMB message since (Friday, 17 June 2011, 11:08 AM) and PIN change has not gone through
Received email notifcation: no messages received in 206 minutes.
- 21:22 reply from Level 1 MR looking further into XML import settings (which are irrelevant)
- 17:55 reply to Level 1 MR that this is related to live import web services
- 17:39 reply from Level 1 MR related to XML import settings (which are irrelevant)
- 16:34 further info sent to MR ticket, The last log in the log file for our production site:
2011-06-17T11:08:04 - LMB Message:enrolment:course id 10860.201109:person id 414619:lmb inserted:enrolled:lmb updated:complete - which seems to indicate that the change that cause this issue occurred between 11:08 and 11:40 Friday.
- 15:40 Message posted to Twitter
- 15:27 MR ticket created
- 15: 16 Setting up ticket, assigning to Grady to contact MR
- 15:13 Rocky restarted LMB, I tested PIN change = same result. Works on Test, not on Prod.
- 14:40 tested PIN change on test server and it works. Production (VH) still does not. Will wait a bit and re-test, and contact MR if it still isn't working by 4pm.
- 14:38 contacted Rocky
- 14:38 changed BW PIN - confirmed did not change in VH
PIN changes are going through for EXISTING users, however new users are not being set up after a BroncoWeb PIN change as they used to.
- Confirmed via Provost's office that this user was set up for Spring 2011, and that is what prevented the PIN change from setting up the user. As soon as user was set up for Fall 2010, it worked as it was supposed to.
- Confirmed both that PIN changes work for an already existing user and that they are not working for a new user setup.
- Set up ticket # 30151
was not actually an incident.
Confirmed that add/drop from BroncoWeb by student is working via live LMB feed.
Noted that some types of drops are coming through in the XML flat file and others are not. This means that even when re-running the Fall 2010 flat file, certain drops from earlier this Summer are not in the flat file, because the LDI file would not pick up DD's as they're usually deleted from the SFRSTCR table.
Confirmed that add/drop from Banner is working via live LMB feed.
- More info: there is a Banner screen called GORICCR (Integration Configuration Settings) where we define the course statuses that indicate enrollment. Currently these are: AD (Acad Dismissal), DG (Drop Course after 10th Week), DW (Drop Course 5th through 10th week), RE (Registered via Banner), RW (registered via Bronco Web), RS (Reinstated student), SA (suspended attend after 3 weeks), and some other statuses we don't use. All other statuses would be considered not enrolled. If this were changed to only RE, RW, and RS as our enrolled statuses, this may mean the LDI XML file would properly pick up all the unenrolls.
Ticket 29191 reports that there are discrepencies between the Fall enrollments in at least 3 courses between VH and BroncoWeb. There are both users who should have been added or dropped that are not.
- Posting note to VH systems notifications
- Proposed 3 possible scenarios to ticket reporter: 1) DOE runs flat files to correct issues, 2) DOE manually checks add/drops, or 3) instructors check add/drops. DOE prefers method 1 to pick up all/any potential issues sitewide.
- Gathering data to check LMB logs.
Confirmed that Live LMB Web Services feed is working on MR test server
- started up live feed to production and tested PIN change
- ran 201005 and 201009 flat files in test/prod
- Thank Bernie Claus!
flat files for 201005 and 201009 ran on both test and prod, PIN Change worked on both test and production via live feed,
Moodlerooms repaired the ability to run flat file, new flat file was run
Moodlerooms repaired the ability to run flat file, new flat file was run
Flat file process repaired
- Student Accounts said they will stop using DD as a way to unenroll students in the future.
- Flat file XML process was changed so that only RE,RS, and RW as "Enrolled" statuses.
Discovered that some unenrolls were still not showing up correctly in VH.
- research: based on how we're currently set up the following course statuses would be passed to Moodle as still enrolled (i.e. status of '1'):
Other course status' would be passed to Moodle with a status of '0'. Note that courses with a DD (Drop/Delete 1ST Week) status that got deleted from Banner would NOT be included in the full extract file that's being loaded into Moodle. In this case I'm guessing the status would remain as "enrolled" in Moodle.
So, in order to have this process work properly, I'm thinking we'd have to 1) change our setup to consider only RE,RS, and RW as "Enrolled" statuses and 2) not delete from Banner courses with a DD status.
XML folder process not working on MR again.
- received reply that MR will be addressing on 6/8/2010
- sent another query, 6/7/2010 a.m.
- reported to MR 6/4/2010 16:00
Unenrolls now appear to be working after changing Parameter 17 (Extract Deletable Enrollments) to Y.
- Loaded LDI file to production
- Loaded LDI file to test server and confirmed Unenroll messages logged.
- Changed Parameter 17 to Y
Discovered that flat files were not processing "drops" (unenrolls).
- Operations discovered Parameter 17 (Extract Deletable Enrollments) for ICGORLDI was set to N.
- notified Operations
- confirmed via LMB logs that unenrolls hadn't processed since running 201002 data on 5/6/2010
- Live feed still down.
- Loading Summer 2010 flat files daily bet 3-5 p.m. to update all enrollments: adds, drops, etc.
- Have received table data from MR to examine to see if Banner data has changed over time.
- need to examine table data from raw XML stored from LMB live feed.
- need to prune the LMB Tables (over 7 gigs of stored data in some tables).
- Forwarded prunelmbtables.php file to install.
ON THE MEND:
MR discovered a permissions issue on the SFTP server which would most likely explain the lack of logging and processing of the XML flat file.
- Live LMB feed is still not working.
- LMB digester processed the Summer term file via folder, which loaded Summer courses with enrollments, set up Summer users, etc.
- Ran and uploaded new XML file with Summer term
- MR fixed the permissions issue
LMB still down - testing flat file fixes
- 197test server: XML folder (extractprocess.php) took long time to run, but worked for 201002 (current file) as well as for 200802 (older file).
- MR test server: XML folder "waiting for..." times out, basically.
LMB still down - both flat file and live feed at this point.
- ticket to MR
- IM with Eric Merrill about why flat file won't process.
on-going from below
on-going from below...
- If issue continues beyond week before Summer term starts, will load flat files with more frequency (daily at that point.)
- most likely going to have to run manual flat files at certain intervals (weekly) until end of term or the issue is resolved.
- Banner to LMB issue, no data going through
- also noted: enrollments that were added in Moodle (not Banner) may have been lost during the course updates note below, including hidden assignments.
- started new log 5/6/2010 8:30 a.m., but no further messages
LMB noted as no longer setting up new users, PIN changes not going through.
- double checked 5/4 a.m. ,
- also noted: LMB log shows much activity in setting up old courses and enrollments for 200709, 200802 in "Development and Design" category.
- Contacted Eric Merrill, Rocky,
- submitted ticket to Moodlerooms 11:15
- Logs show that old terms started being populated on 5/2/2010 11:55 a.m., users from 5/1 at 05:00, 5/2 at 06:52
- Issue identified as Banner INT Comp module installation on 5/4; older terms still set as active.
- Gordon and Rocky went through setting older terms to uncheck "synchronize partner systems" on soaterm
- Grady set up new "dev and design" category, moved sub-categories, renamed the current category to "LMB Issue" and moved it to bottom to isolate the unneeded courses.
- 05/05/10, 06:00 a.m. status: still processing course, enrollments. Log since last night 21:30 shows work on terms 200809, 200805, 200902.
- The log also shows that it "created course" for courses that don't exist, and "updated course" for those that do.
- Located courses from the log that were "updated" to verify if the course was overwritten. It appears that those that were "updated" were not overwritten and the original course content, students, grades, etc. are in place based on these courses from the log: 10110.200902, 10137.200902, 10139.200902.
- PIN changes and new users continue to not be working until the above process is complete.
- Above process completed 03:55 a.m. on 5/6/2010, however no further messages have been received. (to be continued above)
The issue with new users continues, and tests yesterday established that new user PIN changes are not hitting the LMB log. The plan is to continue working with Moodlerooms toward resolution, and attempt a manual LMB process today to see if the new users are set up through the manual process.
upgraded to newer version 0.8.3 of LMB digester.
04/23/10 p.m. PIN changes in BW now result in pw changes in VH (already existing users) or new users being set up in VH.
Users started reporting that a PIN change would not set them up as a user in Moodle. This was confirmed and reported to Moodlerooms and Eric Merrill on 4/5/10, and a Moodle Tracker issue established on 4/13/10 ([http://tracker.moodle.org/browse/CONTRIB-1974)
On-going, MR still working on it.
Another Banner XML dump loaded into LMB Digester.
02/04/10 11:45 call from MR - resolved?
11:50 LMB back up and confirmed working via PIN change.
LMB still down due to SSL error. On-going talks with Moodlerooms who are in contact with Certificate Authority.
Manual Banner XML dump loaded into Vancko Hall. This may result in PIN changes going through and enrollment changes. Some courses may have lost "hidden assignments" in the instructor roles due to the manaul process.
LMB is noted as down due to SSL errors that began in the logs on Saturday. This was following the reports from users of receiving the "untrusted browser" notice when attempting to log in to Vancko Hall.
Reported to Moodlerooms. After some research, they believe the SSL cert. is corrupt and that the Certificate Authority changed something about the cert. without notifying them.
LMB is noted as not working after call from new faculty member, double checked by csn.
Additional info: Ticket 24908 - a student changed PIN on 11-24-09 - not realizing it would also change PIN in VH. The PIN finally changed sometime during night of 12/4/09, so LMB was down since at least 11-24-09
messages were backed up to capacity, restarted LMB, and they are now moving through.
Update: 12/04/09 - 10:08 a.m. - LMB still backed up - estimated time of all messages to move through is about 10 p.m. 12/4/09
ticket #22943 - a student enrolled in several courses in Banner, not showing up as enrolled in the courses in Moodle. Checked LMB log to find several "person not found in moodle" errors related to this user. Checked previous logs and found 449 other instances of the "person not found" error relating to other user enrollments in courses. Note, however, that we have not received any other related tickets at this point. Uploaded all error logs to ticket.
Possible the user was deleted during Spring cleanup?
Possible the LMB has some sort of error/malfunction?
More research necessary.
Note: it's possible that students are showing as enrolled in courses in Banner, and not in VH, due to this. If a user is confirmed enrolled in Banner, user may immediately be added to course in VH using "Assign Roles" by either instructor or Service Center.
several reports of changed PINs not going through. LMB was confirmed to be queing up messages and not sending them.
LMB was restarted, and the messages are now going through again. There were 7,000 some messages, so it will take a couple hours for them all to go through.
9/2/09 back to 0 queued messages by 15:55
backed up, down. (student contacted HD, had changed PIN in BW, works in BW, Banner, not in VH)
working on server , resolved
-----back to 0
by 14:29 !
backed up due to Moodlerooms maintenance causing no access
waiting for Moodlerooms to reset access
1900 as of 8/19/08, 9:07 a.m.
9: 40 a.m.
Moodlerooms notified us of a new IP address which is likely the cause of this issue
ticket to networking to change IP address
LMB was noted as down, cause was found to be WebCT backup issue
- ticket 17616 noted that LMB is down.
- (LMB latest log entry (Moodle-side) at 5:57 a.m. 9/29/08)
- Confirmed down. SysAdmin thinks it has to do with SSL cert renewal.
- MSG sent to Mrooms support.
- Mrooms replied - no change on their end.
- 10/1/08: 9:41 a.m.: SysAdmin restarted LMB - messages now being sent to both WebCT and Moodle.
resolved 10/1/08 at 12:00 p.m.
technician noted that LMB had started queuing messages again, restarted LMB and messages are now being sent again, queue is going down.